{"id":2921,"date":"2021-01-15T10:52:59","date_gmt":"2021-01-15T15:52:59","guid":{"rendered":"https:\/\/www.vintagecomputing.com\/?p=2921"},"modified":"2021-01-15T10:52:59","modified_gmt":"2021-01-15T15:52:59","slug":"reverse-engineering-prodigy-part-1","status":"publish","type":"post","link":"https:\/\/www.vintagecomputing.com\/index.php\/archives\/2921\/reverse-engineering-prodigy-part-1","title":{"rendered":"Reverse Engineering Prodigy, Part 1"},"content":{"rendered":"<p><img decoding=\"async\" src=\"\/wp-content\/images\/prodigy\/prodigy_logo_crop.png\" alt=\"Prodigy Online Service Logo\" class=\"centered shadow2\" \/><\/p>\n<p><em>[ Please welcome Phillip Heller, <\/em>VC&#038;G&#8217;s<em> newest contributor, who is a member of the <a href=\"https:\/\/www.vintagecomputing.com\/index.php\/archives\/1063\/bringing-prodigy-back-from-the-dead\">Prodigy Preservation Project<\/a>. Phillip will post more updates on his progress here in the future. &#8211;Benj ]<\/em><\/p>\n<p>Beginning in the mid 1980s, there were a number of online &#8220;walled gardens&#8221;. Among them were CompuServe, Genie, Delphi, Quantum Link (later PC-Link, AOL, etc), and Prodigy. The latter two were interesting in that they relied on specific client software to access the service.<\/p>\n<p>Quantum Link was certainly novel for the fact that it furnished a graphical online experience for the Commodore 64, and Prodigy was novel for its use of the NAPLPS graphics standard, client-side P-Code virtual-machine, and hierarchical caching. In the early 2000s, some folks nostalgic for Q-Link set out to reverse engineer it, which was a success with <a href=\"http:\/\/q-link.net\/\">Q-Link Reloaded<\/a> launching sometime in 2005.<\/p>\n<p>It&#8217;d be interesting to do a similar thing with Prodigy: To reverse engineer the client and rebuild a mock server with enough functionality to enable those interested to relive another one of the experiences of the early commercial online world. This is the first in a many part series about doing just that.<\/p>\n<p><!--more--><\/p>\n<h3>It&#8217;s in the Patents<\/h3>\n<p>What is particularly interesting about reverse engineering Prodigy is that it was <em><a href=\"https:\/\/patents.google.com\/patent\/US5347632A\/en\" target=\"_blank\" rel=\"noopener noreferrer\">patented<\/a><\/em>, and unlike contemporary patents, the patent is usefully descriptive. When reading the patent and contemplating a reverse engineering effort, I was very surprised to read the following:<\/p>\n<blockquote><p>The source code for RS 400 is provided as part of this specification. This source code can be found in the application file and is incorporated herein by reference.<\/p><\/blockquote>\n<p>So, with this knowledge, I sought out the patent application on the USPTO Public PAIR system and retrieved 2,218 pages of pre-ANSI C source code that was probably printed on a dot-matrix printer, is riddled with various patent office &#8220;received&#8221; stamps and other such things, and has a fair bit of noise precluding an easy OCR job. I have run it through OCR and produced a searchable PDF, which is somewhat helpful &#8211; more on this later.<\/p>\n<h3>The Steps Ahead<\/h3>\n<p>So, with the patent text and source code in hand, I set out to start the reverse engineering effort. Before I get too far into where I&#8217;m at, a quick description of how the Prodigy client worked:<\/p>\n<ol id=\"block-d2ba80cd-fc9e-4325-826a-de7f0256a0de\" class=\"block-editor-rich-text__editable block-editor-block-list__block wp-block is-selected is-typing rich-text\">\n<li>The client (the &#8220;reception system 400&#8221; in the patent) would be executed, would read the file <code class=\"preserve-code-formatting\">CONFIG.SM<\/code> wherein it would find many configuration options including &#8220;object&#8221;, which would specify the first <strong><code class=\"preserve-code-formatting\">Page Template Object<\/code><\/strong>.<\/li>\n<li>That first object would be interpreted by the client and result in the display of a login screen, definition of fields for user id, password, and would bind some logic to field events through a <strong><code class=\"preserve-code-formatting\">Program Call Segment<\/code><\/strong>.<\/li>\n<li>The <code class=\"preserve-code-formatting\">Program Call Segment<\/code> would invoke an application written in a language called <strong><code class=\"preserve-code-formatting\">TBOL<\/code><\/strong> for <em>Trintex Basic Object Language<\/em> (later <strong><code class=\"preserve-code-formatting\">PAL<\/code><\/strong> for <em>Prodigy Application Language<\/em>; see an example <a href=\"http:\/\/www.info.univ-angers.fr\/~gh\/hilapr\/beers\/schade\/p.html#PAL\" target=\"_blank\" rel=\"noopener noreferrer\">here<\/a>). Such applications were compiled into binary form that was executed by a virtual machine implemented in the client (look for the function <strong><code class=\"preserve-code-formatting\">tboldrv(...)<\/code><\/strong>)<\/li>\n<li>The program ultimately called as a result of providing credentials and clicking &#8220;login&#8221; likely invoked the TBOL verbs <strong><code class=\"preserve-code-formatting\">connect<\/code><\/strong>, <strong><code class=\"preserve-code-formatting\">send<\/code>,<\/strong> and <strong><code class=\"preserve-code-formatting\">receive<\/code><\/strong> among others. The content of the initial <code class=\"preserve-code-formatting\">send<\/code> was some sort of login request, and the content received from the server by the call to <code class=\"preserve-code-formatting\">receive<\/code> was likely some sort of login response.<\/li>\n<\/ol>\n<p>So, with that out of the way, here has been my approach to reverse engineering Prodigy:<\/p>\n<ol id=\"block-b3c07617-b438-4530-ae64-6be46f5d1cea\" class=\"block-editor-rich-text__editable block-editor-block-list__block wp-block is-selected is-typing rich-text\">\n<li>Get a client working. I&#8217;ve started with a version of the Windows client running under Windows 95 within qemu, which was configured to connect to tcpser, which was further configured to connect to my mock server. I chose this combination because older DOS clients fail when they don&#8217;t receive an expected response to the Hayes <code class=\"preserve-code-formatting\">ATI<\/code> command. I&#8217;ll come back and do PRs to various projects to implement this. \u00a0Subsequent conversation with Jason Overland reveals that\u00a0<code class=\"preserve-code-formatting\">modem type:NULLMOD<\/code> in <code class=\"preserve-code-formatting\">CONFIG.SM<\/code> will remove the need for the intermediate modem emulation.<\/li>\n<li>Get the client connecting to some mock server. I&#8217;m using Python for the time being for its suitability in prototyping.<\/li>\n<li>Implement the <code class=\"preserve-code-formatting\">DIA<\/code> protocol serializer and deserializer as described in the Patent and aided by the source.<\/li>\n<\/ol>\n<p>I&#8217;ve done this much, and I could see a <code class=\"preserve-code-formatting\">DIA FM0<\/code> header within the initial bytes sent by the client, but further encapsulated. I initially missed this in the Patent, but it is described as follows:<\/p>\n<blockquote><p>Link communications manager 444 is subdivided into modem control and protocol handler units. Modem control (a software function well known to the art) hands the modem specific handshaking that occurs during connect and disconnect. Protocol handler is responsible for transmission and receipt of data packets using the TCS (TRINTEX Communications Subsystem) protocol (which is a variety of OSI link level protocol, also well known to the art).<\/p><\/blockquote>\n<ol id=\"block-63f58c5d-7fa7-4510-bc50-3f322b777ca9\" class=\"block-editor-rich-text__editable block-editor-block-list__block wp-block is-selected is-typing rich-text\" start=\"4\">\n<li>Implement the <strong><code class=\"preserve-code-formatting\">TCS<\/code><\/strong> encapsulation. \u00a0This was interesting in that I found the CRC tables in the assembly to have bytes obstructed by aforementioned Patent Office artifacts. \u00a0I was able to find the corresponding bytes in the binary, but it was just as easy to search the web for a few of the bytes, which reveals it is the CRC-16\/X.25 that is helpfully implemented in the python\u00a0<a href=\"https:\/\/pypi.org\/project\/crccheck\/\">crccheck<\/a>\u00a0module.<\/li>\n<\/ol>\n<p>So, at this point I have enough to properly de-encapsulate the login request, and even send back a response that has a properly calculated CRC. But, the login request is an &#8220;application message&#8221;, likely a data structure passed to that <code class=\"preserve-code-formatting\">send<\/code> verb I mentioned earlier.<\/p>\n<p><a href=\"https:\/\/www.vintagecomputing.com\/wp-content\/uploads\/2021\/01\/a-start....png\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.vintagecomputing.com\/wp-content\/uploads\/2021\/01\/a-start...-450x210.png\" alt=\"Prodigy reverse-engineering progress.\" width=\"450\" height=\"210\" class=\"centered shadow size-large wp-image-2927\" srcset=\"https:\/\/www.vintagecomputing.com\/wp-content\/uploads\/2021\/01\/a-start...-450x210.png 450w, https:\/\/www.vintagecomputing.com\/wp-content\/uploads\/2021\/01\/a-start...-200x93.png 200w, https:\/\/www.vintagecomputing.com\/wp-content\/uploads\/2021\/01\/a-start...-400x186.png 400w, https:\/\/www.vintagecomputing.com\/wp-content\/uploads\/2021\/01\/a-start...-768x358.png 768w, https:\/\/www.vintagecomputing.com\/wp-content\/uploads\/2021\/01\/a-start....png 1024w\" sizes=\"auto, (max-width: 450px) 100vw, 450px\" \/><\/a><\/p>\n<p>So, now there are some additional steps:<\/p>\n<ol id=\"block-a5d019d3-cbdb-407b-b58a-d16815d84884\" class=\"block-editor-rich-text__editable block-editor-block-list__block wp-block is-selected is-typing rich-text\" start=\"5\">\n<li>Transcribe more of the source related to the TBOL virtual machine. Interestingly, the source does include some interactive debugging and disassembly code, sadly all conditioned on a <code class=\"preserve-code-formatting\">#define<\/code> that was certainly unset for compilation of the distribution artifacts. \u00a0The transcription is necessary because it is simply too difficult to jump back and forth in the PDF, even with marginally searchable text based on poor OCR results.<\/li>\n<li>Build a disassembler for the TBOL binaries, based on introspection of the TBOL opcodes and per-opcode operands.<\/li>\n<li>Determine exactly what structure the <code class=\"preserve-code-formatting\">recv<\/code> call in the login process was reading into. As the code lacks debugging symbols, it&#8217;ll probably be necessary to disassemble more TBOL programs to understand the uses of that structure such that a proper response can be constructed.<\/li>\n<li>Disassemble yet more TBOL programs to implement more capabilities; e.g., the <code class=\"preserve-code-formatting\">STAGE.DAT<\/code> file that existed on the client installation media was pre-warmed with certain content, like the login objects described above as well as other content expected to be popular &#8211; like weather maps and forecasts, news headlines, etc. Some of these might be possible to recreate with current content.<\/li>\n<\/ol>\n<h3>A Couple Shout-Outs<\/h3>\n<ol id=\"block-cd64e5f6-7537-47b4-9875-c2497b4d2553\" class=\"block-editor-rich-text__editable block-editor-block-list__block wp-block is-selected is-typing rich-text\">\n<li>Benj Edwards, Jason Overland, and Jim Carpenter, the latter of whom built the <a href=\"https:\/\/github.com\/jim02762\/prodigy-classic-tools\" target=\"_blank\" rel=\"noopener noreferrer\">prodigy-classic-tools<\/a> which parse <code class=\"preserve-code-formatting\">STAGE.DAT<\/code> based on a reading of the patent and code as well.<\/li>\n<li>The <a href=\"https:\/\/construct.readthedocs.io\/en\/latest\/\" target=\"_blank\" rel=\"noopener noreferrer\">python-construct<\/a> library, which makes quick prototyping pretty easy. I ultimately went another way, but I did re-implement some of the <code class=\"preserve-code-formatting\">STAGE.DAT<\/code> parsing with this, and it is certainly a more fluent way to describe such structures.<\/li>\n<\/ol>\n<p><em>[ For more info on the Prodigy Preservation Project, <a href=\"https:\/\/www.vintagecomputing.com\/index.php\/archives\/1063\/bringing-prodigy-back-from-the-dead\">check out this post<\/a>. If you&#8217;d like to help the effort in transcribing the source code, let us know. We also have a working Slack group dedicated to this project for those interested in joining the effort. &#8211;Benj ]<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>[ Please welcome Phillip Heller, VC&#038;G&#8217;s newest contributor, who is a member of the Prodigy Preservation Project. Phillip will post more updates on his progress here in the future. &#8211;Benj ] Beginning in the mid 1980s, there were a number of online &#8220;walled gardens&#8221;. Among them were CompuServe, Genie, Delphi, Quantum Link (later PC-Link, AOL, [&hellip;]<\/p>\n","protected":false},"author":22,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7,6,35,10],"tags":[1698,659,859,689,1701,1697,385,1450,1700,1699,1702],"class_list":["post-2921","post","type-post","status-publish","format-standard","hentry","category-computer-info-and-history","category-computer-projects","category-internet-history","category-computers","tag-jason-overland","tag-modems","tag-online","tag-online-services","tag-patent","tag-phillip-heller","tag-prodigy","tag-prodigy-preservation-project","tag-q-link","tag-reverse-engineering","tag-source-code"],"_links":{"self":[{"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/posts\/2921","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/comments?post=2921"}],"version-history":[{"count":9,"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/posts\/2921\/revisions"}],"predecessor-version":[{"id":2932,"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/posts\/2921\/revisions\/2932"}],"wp:attachment":[{"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/media?parent=2921"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/categories?post=2921"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.vintagecomputing.com\/index.php\/wp-json\/wp\/v2\/tags?post=2921"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}