XML Diff and Merge

Anatol Belski

Saturday, October 12. 2013

XML Diff and Merge

Excited by the the PERL module XML::DifferenceMarkup I've decided to make the same possible in PHP. The new extension xmldiff recently arrived at PECL. It utilizes the libbdiffmark, the library behind XML::DifferenceMarkup. The xmldiff extension can operate on local XML files, strings in memory or DOMDocument objects.
Imagine you have a XML data source delivering periodical updates. That updates have to be incrementally imported into some project. The most practical example from the real life were a product catalog in some web shop. The usual way to achieve the goal in PHP were to parse the incoming XML and compare it with existing data. Or even just to repeat the whole import. Depending on the data size that operation may utilize a lot of memory, CPU circles and possibly bandwidth. Now, with the xmldiff extension, the task becomes trivial:

  • step one - create the diff from the source and target XML documents
  • step two - apply the diff to the source XML document


In between the diff document can be reviewed as it's in human readable format.

Here's the example of how it works.

File from.xml, the data we have in the project right now
XML:
<?xml version="1.0" encoding="ISO-8859-1"?>
<breakfast_menu>
    <food>
        <name>Belgian Waffles</name>
        <price>$5.95</price>
        <description>Two of our famous Belgian Waffles with plenty of real maple syrup</description>
        <calories>650</calories>
    </food>
    <food>
        <name>Strawberry Belgian Waffles</name>
        <price>$7.95</price>
        <description>Light Belgian waffles covered with strawberries and whipped cream</description>
        <calories>900</calories>
    </food>
</breakfast_menu>
 


File to.xml, the incoming data update
XML:
<?xml version="1.0" encoding="ISO-8859-1"?>
<breakfast_menu>
    <food>
        <name>Belgian Waffles</name>
        <price>$6.35</price>
        <description>Two of our famous Belgian Waffles with plenty of real maple syrup</description>
        <calories>650</calories>
    </food>
    <food>
        <name>Strawberry Belgian Waffles</name>
        <price>$7.95</price>
        <description>Light Belgian waffles covered with strawberries and whipped cream</description>
        <calories>900</calories>
    </food>
    <food>
        <name>Berry-Berry Belgian Waffles</name>
        <price>$8.95</price>
        <description>Light Belgian waffles covered with an assortment of fresh berries and whipped cream</description>
        <calories>900</calories>
    </food>
</breakfast_menu>


It's to see that the first node differs in both XML documents only with it's price, the second node is the same in both. Additionally the second document has a new third node. Lets use the xmldiff extension to produce the diff

PHP:
<?php
 
$zxo = new XMLDiff\Memory;
 
$from = file_get_contents("from.xml");
$to = file_get_contents("to.xml");
 
$diff = $zxo->diff($from, $to);
 
$merge = $zxo->merge($from, $diff);
 
echo $diff, "\n\n", $merge;


And that's how the diff XML document looks like
XML:
<?xml version="1.0"?>
<dm:diff xmlns:dm="http://www.locus.cz/diffmark">
  <breakfast_menu>
    <food>
      <dm:copy count="1"/>
      <price>
        <dm:delete>$5.95</dm:delete>
        <dm:insert>$6.35</dm:insert>
      </price>
      <dm:copy count="2"/>
    </food>
    <dm:copy count="1"/>
    <dm:insert>
      <food>
        <name>Berry-Berry Belgian Waffles</name>
        <price>$8.95</price>
        <description>Light Belgian waffles covered with an assortment of fresh berries and whipped cream</description>
        <calories>900</calories>
      </food>
    </dm:insert>
  </breakfast_menu>
</dm:diff>


The merge document contents will be our target document.

And that's actually it. Now only the diff data can be delivered for updates, which will do the import operation much faster and provident. That means also better reliability based on the reduced sources of error. The human readability makes it much easier to review the changes manually.

Many thanks to Václav Bárta for creating the libdiffmark and bringing it into public domain.
Many thanks to Remi Collet for the extension improvements. Also, the Fedora RPMs can be fetched fro his repository.

Links:

http://search.cpan.org/~vbar/XML-DifferenceMarkup-1.04/lib/XML/DifferenceMarkup.pm
http://pecl.php.net/xmldiff
http://penguin.cz/~vbarta/diffmark/
http://rpms.famillecollet.com/
Posted by Anatol Belski in C/C++, Perl, PHP5 extentions, PHP7 extensions, Web at 21:54 | Comments (4) | Trackbacks (0)

Trackbacks
Trackback specific URI for this entry

No Trackbacks

Comments
Display comments as (Linear | Threaded)

Dear Colleagues,

I have found some issues in this package. I'm using xmldiff package 0.9.2.5.4 ; PHP 5.4.17 Apache 2.2.25.
For example I have two xml files:

File "from.xml" contains:



321
123



File "to.xml" contains:



123
321



//As I'm using code:

$zxo = new XMLDiff\Memory;

$dir1 = file_get_contents("upload_IR21/from.xml");
$dir2 = file_get_contents("upload_IR21/to.xml");

$diff = $zxo->diff($dir1, $dir2);

/*
$zxo = new XMLDiff\Base;

$dir1 = dirname(__FILE__) . "/upload_IR21/from.xml";
$dir2 = dirname(__FILE__) . "/upload_IR21/to.xml";
*/

$diff = $zxo->diff($dir1, $dir2);
$file = 'differences.xml';
file_put_contents($file, $diff);

//I get result in differences.xml file:










321





Could you please inform me from where the




comes?

Also please kindly inform me if there is a method which differs two xml files without matter of xml nodes order?
#1 Paulius on 2014-02-26 12:37 (Reply)
Hi Paulius,

unfortunately the blog cuts out all the XML stuff you posted, such a sanitizer junkie :-)
But from what I can see you're probably concerned about something you see in the diff. Please note that the diff document contains instructions for syncronizing document A to document B. The node order matters as otherwise we wouldn't know how to merge it.

Some XSD schema support were nice to have, whereby i'm still concerned it could lead to the ambiguity in some cases. Please feel free to put such a feature request under bugs.php.net .

Thanks.
#1.1 Anatol Belski on 2014-02-26 13:08 (Reply)
Dear Anatol,

Yes I see that a lot xml stuff were cutted out from comment. Also I have made some type errors. As I want you to know all the issue, please find it posted clearly in stackoverflow. Here is the link:
http://stackoverflow.com/questions/22042195/xmldiff-issues-on-php

Thanks a lot for your comments!
#2 Paulius on 2014-02-26 14:17 (Reply)
Hi Anatol,
Just a quick question, is there any specific way to have the difference file also delete the unchanged records?

So it would compare file a with file b, then merge file a with the difference file. But the difference file will contain the following:

- Adding of new records
- Updating of changed records
- Deletion of unchanged records

Thanks!
#3 Carl on 2014-08-21 10:12 (Reply)

Add Comment

Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
BBCode format allowed
BBCode format allowed
 
Submitted comments will be subject to moderation before being displayed.
 

Quicksearch

Categories

Archives