https://github.com/paulgb/simplediff/blob/master/python/simp...

Quick question, is this the sort of technique that used for general text diffing? I see you effectively tokenize the string using whitespace to split and then pass in 2 lists of tokens. Is that the way it's generally done, and you just choose a suitable tokenizer for your specific use-case?

It's used for updating Chrom{e,ium)

https://www.chromium.org/developers/design-documents/softwar...

  in the same way that one-sentence-per-line  
  gives you much more fine-grained diffs than  
  one-paragraph-per-line, notching it down to  
  one-phrase-per-line gives the best results.

  a simple javascript routine can do that split,  
  and then rejoin the lines after you do a diff.  

  i've written this up extensively, to no notice.  

  http://zenmagiclove.com/simple/breaker.html

  https://github.com/bbirdiman/breakerbreaker

  ***

  i've since found that a dozen changes suffice:

  replace ". "  -- with --  ". \n"
  replace ", "  -- with --  ", \n"
  replace "? "  -- with --  "? \n"
  replace "! "  -- with --  "! \n"
  replace ": "  -- with --  ": \n"
  replace "; "  -- with --  "; \n"
  replace ") "  -- with --  ") \n"
  replace "] "  -- with --  "] \n"
  replace "} "  -- with --  "} \n"
  replace "-- "  -- with --  "-- \n"
  replace "' "  -- with --  "' \n"
  
  replace '" '  -- with --  '" \n'

  ***

  and, of course, to revert those changes,
  you simply change " \n" to " " and boom.

Almost entirely.

Most programming diff algorithms operate on a line-by-line basis. That means that a single character change in a line marks the whole line as changed. For prose, a single 'line' is usually a paragraph, so it's pretty obvious why they don't work well. You might check out gnu `wdiff` for what a word-by-word diff could look like. I haven't really looked into the area deeply, so I don't know what the state of the art is.

Tree diff algorithms definitely exist (I know React.js uses diffs on a virtual DOM to minimize operations), but I'm not sure what the state of the art is for those.

Tree diff is harder because the range of operations is bigger.

React.js cheats by insisting on keys for arrays so matching is much easier.

My specific use case was a document tweaked for two different audiences, and I've been able to make changes in one version and rebase the other version on top of it relatively easily.

From the simple perspective, using a fixed width line in a prose markup language that is mostly whitespace agnostic like Markdown creates okay diffs in a line-based diff tool.

I did some work with tokenized diffs: https://github.com/WorldMaker/tokdiff

That particular tool/experiment uses the Pygments tokenizer used for syntax highlighting and produces interesting somewhat semantically meaningful code diffs. I think the same principles would apply if you used something like a part of speech tagger on prose.

From a different approach, I put some effort into better line-based and file-based diffs of Inform 7 which is a prose format that is less whitespace agnostic than Markdown and also built as a single monolithic file, by converting it to an intermediate format.

The code that does this is here: https://github.com/WorldMaker/APrincessOfMoons/blob/master/i...

That works by splitting the file at things that resembles headers, converting newlines to pilcrows (the paragraph symbol), and essentially reformatting to more of a fixed width format. (All of which is trivially reversible.)

You can see the commits in that repository as an example of what the intermediate format looks and diffs like.

That format handler is written as a plugin to a venerable tool I wrote called musdex: https://github.com/WorldMaker/musdex

I wrote that to better source control zip files as the contents of their zip rather than a binary blob, by letting the zip/unzip operations be automatable as a part of source control operations (pre- and post-commit hooks). This I've used for some of the modern file formats like .docx which are built as zip files of XML files and other assets. For instance, you want write the document in Word, interacting with the .docx, and source control its XML contents. That too gives more useful diffs than source controlling the .docx on its own.