Jump to content

Photo

Unreadable BMP files


  • Please log in to reply
29 replies to this topic

#16
lu9

Actually is ingame, when I went on halloween mode it appeared... nothing special...

#17
shinyquagsire23

shinyquagsire23

    Newbie

  • Members
  • Pip
  • 7 posts

Sorry to bring up a dead thread like this but I recently had the inspiration to recreate LEGO LOCO using the original game files (as opposed to manually ripping textures and stuff). So I did some research, got the RFD and RFH sorted out and now I'm stuck on getting these pesky BMPs to open. Luckily I happen to have some experience in reverse engineering (specifically Pokemon games for the GBA) so opening up a hex editor to take a peek at the files is nothing new.

 

So first off, these files are definitely compressed in some sort of form. If you open the files as a raw image using GIMP you'll see something similar to this:

Please Login or Register to see this Hidden Content

 

Whereas if it were just a bitmap stripped of it's header it would look a bit like this guy from the Yoda Stories data file that I happened to open which comes from around the same time (1990's):

Please Login or Register to see this Hidden Content

 

I'll keep looking but it seems that we're going to have to keep looking. It does seem to have some sort of a header, I believe it's about 8 bytes long since everything else in the file is shifted by 8 bytes. Some notable offsets after looking through some files:

  • 0x0 - Some sort of header

  • 0x408 - Some sort of data, perhaps a palette? The openable files are indexed bitmaps with a 256 color palette, making the images 8bpp images as opposed to something like 4bpp.

  • 0x808 - Compressed (maybe) data. Still need to find compression format. I'm placing my bets on Huffman, LZ77, or ZIP. If we're not so lucky it could be a home-made compression format which might take time to decipher.

Also, I found an interesting file in the buildings folder. It's in reguards to the telephone: Instead of the usual BMP and DAT files, this one has a BMP, DAT, and a BUT file. The BUT file is in the exact same format as the BMP, so this might aid in finding the original format which we already have found to not be BMP. Also, if anyone has anything interesting to contribute I would appreciate it. Hopefully I can figure out this format soon so I can make a data extractor for my game so that I can use the original resources without having to resort to blatantly stealing IG's work. But, if needs be, I will end up ripping the sprites manually and maybe throw in a check for a valid installation of LOCO.

 

EDIT: Also while going through in IDA I found a debug string that says "Error while loading db bitmap" and one that says "LOCOBITMAP Convert - failed to create surface". Perhaps another clue as to what the file format does? It does appear to be converting something so I might be able to identify the compression.

 

EDIT again: Another small breakthrough, but it seems that the developers did in fact implement a function that would open files uncompressed, which will help debugging immesely. Since LOCO was (for the time) quite large I'd imagine that they would have it default to the uncompressed files, and if they couldn't be found it would try and open the compressed files.

Please Login or Register to see this Hidden Content

It's cut off but the full file path is C:\Program Files\LEGO Media\Constructive\LEGO LOCO\ART-res\toybox\newtoy\handle.bmp, which is definately not inside it's compressed file. This code might not fully work though so I'll test it by removing the archives and seeing how it goes.

 

One more EDIT: So apparently it does need the data files due to the fact that it will try and read from them on startup. I'll have to see which it tries to read from first and get the results back to you guys. On another topic, I found one file that has both a weird formatted file and a normal bitmap, so we might be able to use that to figure out how it's compressed. The two files are roads/tjunct-swint.bmp and (here's that extention again...) roads/tjunct-swint.but. I'm pretty sure the .but is the original file extension before they went in and renamed them to all be BMP's, so if we can figure out this file then it blows the doors wide open for modding. Maybe. That and it allows me to view all the images which is why I'm here.


  • jamesster, McJobless, le717 and 1 other thanked this

#18
McJobless

-an inspiring tale that made many cry, fed the ducks, saved the whales and brought Abe Lincoln back alive to say "This play is quite enjoyable, I hope I don't get sh..."-

Holllllly s***. A guy who likes Sam & Max AND knows his way around a Hex Editor. I like this very much. Keep it going, mate, you've done really well so far.

#19
shinyquagsire23

shinyquagsire23

    Newbie

  • Members
  • Pip
  • 7 posts

OK, so I've been playing with the bitmaps (AKA flooding them with random bytes and values) and I've come to the following conclusion: These bitmaps are really really really weird.

 

I was pretty much wrong about everything in my initial analysis. However, there are some things I'm glad to be wrong about. First off, I've discovered why some of these are formatted as normal bitmaps and why some of them aren't: animations. Every one of those files has an animation. Just take a look in the buildings folder, only the ones with multiple frames will have an unreadable file.

 

Second, I found some interesting animations for the telephone. It turns out that there is also a blue and red version of it that was never used, as well as some unused easter eggs. If I can I'll try and get a pic up for you guys to see.

 

Third, these graphics are indeed bitmaps, but they are very very very weird ones at that. Instead of being like a normal bitmap they start their image from the end of the file, so the top right pixel corresponds to the last byte in the bitmap file. And to make things even weirder they put each row of animation one after another, which is why they looked so darn weird in GIMP. So just as an example, instead of having it so that you have two separate images like most games do animation, they have the two images weaved together with the corresponding rows right next to each other. Weird. But, I am very happy because this means that these are both readable and editable. It's just a matter of figuring out the rest of the file. Also, I've discovered that the preview (with the magnifying glass) image is separate from the one displayed.

 

And lastly, I found the purpose of the .but files. They are used in the toolbox as previews, because if you look carefully they are not just scaled versions (well, most aren't. The telephone is.), but in fact separate images.

 

I'll keep looking into this. I'm actually very excited now that I know that it's possible to edit these easily (ie without cracking a compression algorithm). All that's left is to find the palette inside the file as well as the other information (the number of frames and the size of the image just as an example). Once we have that we can start converting these to editable bitmaps and then just use those for the purpose of modding.

 

And since it seems like you guys didn't know this before (based on the top post here), those .dat's are just text files. They only look weird if you open them in either a

  • Notepad
  • or a Hex Editor

If you open them in wordpad you'll find them to be perfectly editable. I'm a bit curious to see if LOCO dynamically scans for those .dats and if they'll show up in the toolbox if you copied a building or something.

 

EDIT: Number of frames is specified in the .dat file. Also, it doesn't dynamically scan, which means there's a list of files for it to open out of the buildings folder. If I can get another building to be added this could mean a lot in terms of modding. Also, it seems that the .WAVs have the same exact header as the .BMPs, which is interesting. I'm guessing it's just raw PCM data.


  • jamesster, McJobless, le717 and 3 others thanked this

#20
jamesster

jamesster

  • Moderator
  • PipPipPipPipPip
  • 2,509 posts
Please keep posting. You're awesome.
  • McJobless thanked this

#21
WillKirkby

Third, these graphics are indeed bitmaps, but they are very very very weird ones at that. Instead of being like a normal bitmap they start their image from the end of the file, so the top right pixel corresponds to the last byte in the bitmap file.


Actually, "real" bitmaps use this raster order too.
Rather than use "standard" raster order (that being left-to-right, top-to-bottom, like words on a page), they use vertically inverted raster order. It still goes left-to-right, but the y component is flipped.
 

FuBkIke.gif



#22
shinyquagsire23

shinyquagsire23

    Newbie

  • Members
  • Pip
  • 7 posts
 

Please keep posting. You're awesome.

 

Thanks, This is my first time reversing an x86 executable so this is kinda exciting for me.

 

 

 

Third, these graphics are indeed bitmaps, but they are very very very weird ones at that. Instead of being like a normal bitmap they start their image from the end of the file, so the top right pixel corresponds to the last byte in the bitmap file.


Actually, "real" bitmaps use this raster order too.
Rather than use "standard" raster order (that being left-to-right, top-to-bottom, like words on a page), they use vertically inverted raster order. It still goes left-to-right, but the y component is flipped.
 

FuBkIke.gif

 

Didn't know that. In the GameBoy Advance (which is what I usually mess around with) it does it from top to bottom, but in 8x8 squares. Interesting to know that this isn't anything new.

 

 

 

OK, so apparently this in-built bitmap format is a lot more complicated than I thought. Turns out it is slightly compressed, but not using a compression algorithm. Instead they have a system to minimize the number of bytes used which makes things a bit more difficult. Basically, each byte can be as little as 2 pixels to as many as 4, which complicates things since that means that somehow it's selecting other colors from the 256 color palette shown in the normal images, which also means that I have to decipher a crapton of other stuff in there.

 

Aside from that, I did figure out a way to dump images without directly decoding it. In the .dat files there is a property called total_number_of_frames that can be adjusted. So if you take a multi-framed object and set it to 1 frame it will turn up like this when you place the object down, exposing the entire image:

Please Login or Register to see this Hidden Content

So I managed to dump some of the images, but unfortunately they don't exactly insert back in very well (hence the really crappy looking factory above), so we still have work do to. But they do load and animate correctly, it's just that they don't load the proper palette. So if we can figure out the proper palette we could most likely fix the issue. Not to mention that the building .but's will also have to be modified or the executable hacked to view them from a normal BMP instead of a .but. I'd personally rather do the latter because the bitmap format caused me much pain trying to figure out and I got absolutely nowhere outside of what I learned in the last post.

 

Just to give you guys an idea of what we're working with here, I'll throw up some of the ripped animations I got:

factory1_zps0c8062ea.png

telephone_zps5ab4e97a.png

resturnt_zps8f470968.png

Also, this can work to our advantage in terms of other things since we can rearrange the images to rip other items that aren't buildings. So if we can figure out a way to get the palette to work correctly and to load other buildings we could have a potential modding scene here. I personally plan on using the resources to make a sort of "Open LEGO LOCO" which can recreate the game with proper networking support as well as improved modding ability and maybe a more Sim City like engine which could give purpose to the factories and restraunts while also implementing currencies and other things.


  • jamesster, McJobless, WillKirkby and 7 others thanked this

#23
jamesster

jamesster

  • Moderator
  • PipPipPipPipPip
  • 2,509 posts

I want to make love to your posts here.
 

I personally plan on using the resources to make a sort of "Open LEGO LOCO" which can recreate the game with proper networking support as well as improved modding ability and maybe a more Sim City like engine which could give purpose to the factories and restraunts while also implementing currencies and other things.

Oh my goodness yes. Also, one of the things I've always wanted to be able to do in Loco is to directly control a minifigure to explore cities, both my own and others on the network. Just sayin'. :3


  • JimbobJeffers and LimeKiller thanked this

#24
shinyquagsire23

shinyquagsire23

    Newbie

  • Members
  • Pip
  • 7 posts

So I finished ripping all the textures in the buildings folder, and I came across an unused building. I think it's rather interesting:

MbHPoDX.png

So apparently the house that could merge into a double house also was meant to merge into a quadruple house. And since a lot of the .dat files have the easter egg implementations I might be able to make a mod that could restore this since the texture is fully intact, although when it's placed via attaching a .but with the same name part of the texture is cut off, although I'm sure that's fixable. Once I get to ripping the paths folder I might actually find the one beta path that can go over train tracks that everyone seems to be talking about.


  • jamesster, McJobless, le717 and 2 others thanked this

#25
JimbobJeffers

I doubt I can express gratitude as well as jamesster, but this is truly epic. Keep at it.

 

Anyone for a Rock Raiders version of LEGO LOCO?



#26
jamesster

jamesster

  • Moderator
  • PipPipPipPipPip
  • 2,509 posts

Any way to maybe fix the IG tower robot easter egg? Its arms get clipped off.



#27
Tauka Usanake

Tauka Usanake

    Advanced Member

  • Donators
  • PipPipPipPip
  • 307 posts

Any way to maybe fix the IG tower robot easter egg? Its arms get clipped off.

 That would require extending its area as the arms would make the building a 5x3 building from a 3x3, unless you can get overlapping to work and not break things and not break or destroy anything that's taking up those sides including the radar station.

 

Also I need to get on RRU more often as I LOVE THIS.



#28
noghiri

I accidentally disappear for several months, and I miss this!  I'm kicking myself now. xD

 

This has turned out to be way more complicated than I can easily understand... but I'm glad this was accidentally started.

 

Now, to bone up as much as I can on this so I can try to be useful again...



#29
shinyquagsire23

shinyquagsire23

    Newbie

  • Members
  • Pip
  • 7 posts

Well, it seems I've forgotten about this place once again. Sorry for the random disappearance, this was literally just a small one-day project idea which I had and I kinda forgot about it. Although coming back to it, I really want to figure out how this stuff is compressed and stored, as well as the sounds since they seem to use compression as well. I might work on it tomorrow if I can and get to the bottom of this format. If I can get some sort of idea of how this format works by comparing the bitmaps I extracted vs the compressed versions, I might get somewhere.

 

EDIT: Some things I'm noticing after getting a more fresh view on things:

 

The first value in the header seems to be a 32 bit unsigned value, denoting something in terms of length (ie uncompressed size/decoded size). Depending on the file, this value gets marginally larger or smaller (ie the train carts and sounds have much larger values than things like houses and whatnot which only have one frame or many smaller frames)

 

The table at 0x408 seems to be a sort of lookup table, with each value being 32 bits long (4 bytes). The amount of entries in this table from 0x408-0x808 come out to 256 entries, or multiplied by 4, 0x400 bytes. As for what looks up entries in this table, I have a small feeling that the data at 0x808 is just a bunch of byte values referencing this table to make a set of data to be read. Don't have anything official yet, but I do believe that this is a very good hunch considering that while I was messing with the values everything was seemingly random despite the values only being messed with by a tiny bit. This could expain that. It also could explain how this format could be used for sounds as well since this area has been confirmed to not be a palette, but rather a sort of 'palette' of 32 bit values available to use. Again, not entirely sure at this point, but it seems to be a good hunch for now.

 

Once the bitmaps are decompressed, it might be possible that there's another bit of fluff to go through to figure out what format the uncompressed data is in. I'm praying that it's just an uncompressed raw 8bpp bitmap, but with how things are going that might not be the case. I'd also like to test this against sounds as well since it might be easier to figure out the sounds vs the images since sounds are more likely to just be raw data.

 

EDIT 2: I think the table might be in a slightly different format than I thought initially. I'm thinking it might actually be a sort of table which specifies bytes and the number of repetitions of that byte, perhaps in the format <byte> <repetitions> <byte> <repetitions>

 

EDIT 3: I think I'm getting somewhere with this. I'm doing comparisons between the ripped version of house12.bmp I have vs the house12.bmp which is compressed, and it seems to be that I'm somewhat correct. I counted 305 transparent pixels before reaching actual data, and based on the number of 0x55 bytes present at the end of the file, if you counted them and multiplied by 4 you'd get 304. If you add in the other 05 prior to the 55 array it might equal the 305 we're looking for. Now, since each entry in the table is (gasp!) 4 bytes long, we might be able to assume that one of the two hypothesis I had are correct. I'm actually placing my bets on the byte + repetitions because it seems that it would make the most logical sense seeing that every other byte is usually less than 4.

 

EDIT 4: This might be, once again, a little more complicated than I thought. I almost wonder if it's not even a compression format but just an encryption format or something, because this format is used on just about everything from bmp's to wav's even to a file called ee.ini. Honestly at the moment I think that this so called ee.ini might be my best bet on cracking the format since the wav's and bmp's could be under another layer of proprietary formatting, while an INI is a little more easy to guarantee what it will produce. I might actually be able to find a copy of the uncompressed version in RAM while the game runs, or find out how it's uncompressed and go from there. Since it's an INI file it's probably easier to track down than other files.

 

The main issue I've run into is that the data at 0x808 can have values over the length of the table. The table obviously seems to have values used to create the uncompressed data, but I cannot figure out the exact relationship.

 

EDIT 5: Apparently there's an uncompressed EE.ini. Might try to do some comparisons here...

 

EDIT 6: The values used in the table at 0x404 consist only of the values used in the INI, albeit in a slightly different order. However, every value is there.

 

EDIT 7: Not much real progress so far, but, I have found the file loading function in IDA. Apparently the weirdly formatted files are actually supposed to be handled through the resource file extractor. Oddly enough, it was mentioned once by Cyrem, the guy who made the extractor:

 

 

It seems to be marked in the archive file which files have a custom format, so once a finish the next version of this can start working on that. Also, the extraction process of the next version is like 500x faster than v1.1, it completes in like under 30 seconds now.

 

Luckily I actually found the check for compressed and uncompressed files, so now it's just a matter of reverse engineering the function which decompresses the file. If anyone's curious, the file header is formatted as such:

<32 bit word: amount of space to malloc, aka file size> <8 bit byte, something with decompression/compression> <8 bit byte, if 1 file is read as compressed>

All compressed files are normal files underneath, with normal file headers and easy-to-edit stuffs. So once I can write a decompression program you can keep the files uncompressed for as long as you please.

 

EDIT 8: This compression... I literally can't even. It's the weirdest compression I've ever seen in my entire life. It might be because I'm looking at in in assembly but I have no idea how I'm even going to attempt to recompress any of these files. Decompression maybe, but how in the world does this work...

 

EDIT 9: Witchcraft. That is the only answer here. Pure witchcraft.

 

EDIT 10: Success has been had. Documentation and a tool will be coming shortly...

 

EDIT 11: While I'm still kinda messing with things, here's an image file I extracted of the launchpad in it's full frameset:

FDmv5k3.png

This was actually one of the ones I was having difficulty ripping manually. I'm not sure if this is an error with me adding a header to the headerless bitmap, but one of the frames seems to be cut in half on each end for some reason. 


  • McJobless, Tauka Usanake, le717 and 4 others thanked this

#30
Tauka Usanake

Tauka Usanake

    Advanced Member

  • Donators
  • PipPipPipPip
  • 307 posts

Well now I need to get LEGO Loco working on something again. I might have a virtual already loaded for it and Hamachi.

 

I never got to play Loco on Hamachi with others. All the icons work client side I would imagine, right? What if you edited the train sprites? Is it the same train icons if it travels to another's area?