Huggingface GGUF Editor · ggml-org/llama.cpp · Discussion #9268

ggml-org · GitHub

Thanks for the FOSS! It looks nice / useful.
I haven't spent a lot of time looking over the details but one thing comes to mind which might help the UX if it's possible so maybe it is something interesting to discuss or consider as a feature.

GGUFs are usually monolithic up to some size then they're usually but not necessarily split into various usually somewhat equally-large size chunks of up to several/many gigabytes each.

I haven't verified how the file encoding / format is but I assume maybe the metadata is right at the beginning of the first file or at least is a small section somewhere in the group of files.

So if you want to make any kind of metadata change you're by definition modifying a tiny tiny fraction of the typical GGUF file / file-set total.

So it seems easy to inspect / modify that tiny piece online in the HF space UI, but the result is (as I interpret what little I know / see so far) having to download at least a large "chunk" of the (if split) GGUF or the entire single file to obtain a copy of the tiny metadata changes.

If that's true then it might be nice to have SOME way(s) to modify the metadata but any combination of :

A: Generate / download instructions or a script to use the standard llama.cpp local-copy GGUF metadata editor CLI to edit / replace the metadata according to the selected change(s). That way one has only a few lines of code / instruction to download and can apply the desired change without downloading gigabytes of unnecessary GGUF data, too.

B: Generate / download some kind of "binary patch" file / script that can (somehow) just take the binary differences between the original GGUF and the modified one and encode those so one can apply them with any standard (there are plenty of standard / common file patching tools e.g. for unix but also cross platform) patching tool or if there's any reasonable use case for it a custom patch script (which I'd avoid in favor of an already sufficient purpose-built standard patch tool). Then one downloads a tiny set of instructions / patch file and has a straightforward way to apply the differences to a GGUF you already have downloaded via some easy / quick means.

C: Maybe it is already possible but if so it is unknown to me and maybe others so perhaps discussing / commenting on the right techniques would help many users. So let's say there is a possible work flow to instead of downloading the modified GGUF it is actually modified inside or saved to one's OWN (or writable) HF repo / space / whatever; maybe that's where you put a copy of the original GGUF to be modified and you basically want to just make a "small delta" to it to create a new GIT version or whatever that is 99.999% the same except for the metadata block. Ok so then to efficiently download THAT from your modified HF repo / space / whatever where you have a modified copy it would be ideal to use an efficient file transfer / synchronization protocol like rsync or such which will only copy the changed portion (approximately) and realize quickly it does not have to download the rest of the unchanged N GBys. That would involve some kind of efficient binary delta (or git LFS commit condent delta...?) aware transfer protocol or software. IDK how "git lfs pull" or whatever actually might be used optionally or defaults to work in terms of efficiently pulling small updates to large binary files but IF there is a git lfs based way it probably bears promotion / discussion since I may not be the only nescient one. And if HF CLI / UI supports some other "scp", "rsync" or other means to actually efficiently copy small updates of binary large files that'd be great to know about also.
So basically a nice post / blog post / FAQ / whatever might be very useful to many if there
are optimum techniques to "fork" / "clone" some upstream GGUF (from some accessible source repo someone has), copy that into one's own repo without necessary download / upload to your local PC, inspect / edit GGUF metadata and save it to one's HF repo or whatever, possibly re-split the file (see below) then download either the modified original file or the deltas relative to the upstream GGUF highly efficiently SOMEHOW from one's own HF or wherever one has the changes available / saved.

D: It's possibly relevant to the editing workflow but also possibly relevant much more widely to the way ggufs are made / published. So if the metadata "section/chunk" is tiny and the rest of the data is huge then it's nice to be able to edit / send / update it efficiently independently. GGUF already gained the ability to support "splitting" with variable amounts of "data" in each "piece/chunk" file. So if that's so then why not GENERATE a gguf that is SPLIT so that for example somefile-00001-of-00099.gguf contains essentially ONLY the metadata (and maybe other highly relevant small header etc. content) and is tiny. Then the OTHER GGUF split files would get the various pieces of the model data which would be large and highly less likely to need to be edited / updated in small granularity.
IDK of any reason why you / anyone could not intentionally generate "metadata-separate" (or similar effect) GGUFs so you can easily download updates to that one and still have it work with NN other chunks of GGUF file split data which themselves do not (typically) change / get edited in small parts.
So if you're providing an easy way for people to "play with" modified metadata in GGUFs then maybe encoding it (GGUFs) into some "metadata update friendly" split format could be user friendly for
these reasons? Of course I could see promoting the idea to the llama.cpp GGUF encoding / processing / medatata editing tools as well as maybe a nice option / default.

E: Of course I guess it's possible to take a HF space's "software" and if it's open source etc. fork / reuse it etc. So maybe your GGUF editor GUI is already able to be derived-from but anyway the thought crosses my mind that if it is using some simple-ish gradio UI on top of some python or whatever script then maybe there is some nice option to reuse 70% of that or whatever and make a very analogous metadata editor UI people can download / use locally to augment / replace the llama.cpp metadata editing / inspection CLI as a UI option. But I assume that'd turn into a github project or maybe llama.cpp pull request or something to extend the metadata / gguf editing / composition UIs in whatever ways. But maybe something to think about if the user friendly UI UX could be enhanced for local use cases also. I'm not personally proposing a fork or project just sharing a concept FWIW.