PDF to Accessible Markdown: Introducing an Open Source AI Tool

So I'm Blake Bertuccelli-Booth. Here is my, I'm looking for my speaker notes right now, but here they are. I, like Thomas was saying, I'm the Assistant Director of Digital Accessibility Engineering at the University of Illinois Chicago. I am a passionate open source developer.

I've been an open source developer most of my life and accessibility champion. Everything I've worked on in the last several years has been focused on making the internet more accessible to people, not only with disabilities, but people all over the world. And that is kind of what drives most of my work here. I'm also with Dylan Isaac, who we brought on at the University of Illinois Chicago to help build this project and shepherd it into realization.

We're really excited tonight. To be presenting this project that we've been working on for several months. And before that, I think Dylan and I both had the ideas that AI could do this project, but we really pulled the trigger and built it. We are just starting to announce it.

So we announced it publicly at CSUN, a conference a few weeks ago, or was it almost a month ago? Almost. And we're going to be publishing it. So it's a totally open source project.

We're going to be publishing it pretty soon. But we're really excited to introduce it to you. This is only the second time we're publicly introducing this project and the first time in New York. So very excited to be with you all.

So just a quick recap of what we're going to cover. We're first going to look at the problem with PDF accessibility, which some of you might know, some of you not. This is not a highly technical conversation that we'll have. But we will cover the problem with PDF accessibility.

Then we'll go through options other than PDF. Why we need AI. We really tried to not do it with AI for a while, but we definitely need it. And then we'll talk about our solution here.

So just to kind of talk a little bit about the problem with PDFs. PDFs were designed to make documents for printers. That was the initial design doc. And then we've kind of taken PDFs into this portable document that you share around.

It became a really easy way to share documents across different platforms. So if I had a Microsoft Word document and I wanted to share something with somebody who might not have had a Microsoft Word document, I would make a PDF. It became really, really easy to do. And there are even open source libraries that made it even easier to manipulate and change PDFs.

But PDFs, again, were made to transport documents into a printer. They didn't have the same kind of structured code base that you would be familiar with something like HTML. So it was very much a document for the seeing person. And the other big kind of issue behind PDFs is that they are owned by a proprietary company.

Adobe controls PDFs. We worked a lot with different folks around Adobe to find good solutions that could make it accessible. There is a big deadline coming. And I forgot to press next, but that's okay.

I spoke about a lot of this. But there's a big deadline coming April 24th. It's a Title II ADA deadline. And we get it at the University of Illinois, Chicago, and any higher ed organization gets it drilled in their head.

So we decided that we have to come up with a plan to make PDFs accessible by then. Because it's a government mandated time to have that plan in place and to be working towards PDF and document accessibility. And so we see this huge urgency to do that. And that's why we're releasing and working on this project here.

We want to make as many documents accessible. And we want to use this deadline as impetus to make it accessible beyond just an accessibility department. So just to kind of look at what UIC, the University of Illinois, Chicago, which I work with, has. We have a bit over 200,000 PDFs in our system.

Every year we see a lot of, you know, up to 72,000 PDFs uploaded every year. And so the 200,000 is just the tip of the iceberg. But those are the PDFs that we really focus on as PDFs that we have to do something about. Manual remediation that we've been quoted on costs from $5 to $35 per page.

When you have 200,000 PDFs and most of the PDFs are large academic papers with lots of pages, it becomes completely untenable to do manual remediation. So even at the cheapest rate, we're looking at over a million dollars to remediate just those 200,000 PDFs. And that is an economic issue that we just can't work with. We had to find a solution for this.

I, as an engineer, knew that the solution could be in technology. We didn't just have to throw up our hands or bring a bunch of PDFs to the dumpster to light on fire. We could actually use technology to make these PDFs accessible. So when we started to go down this rabbit hole of how do we make the PDFs accessible, our first thought and what we were seeing from a lot of vendors was, let's turn a PDF into an accessible PDF.

And so there was all sorts of different solutions that we were shown. One of the major solutions just basically tagged the PDF. When you're making a PDF accessible, you're tagging the different areas of a PDF and saying what they are so a screen reader can hopefully read them out. And so there are AI solutions that will try to auto-tag PDFs.

But even the best solutions that everybody said was working for them, when we tested it, it didn't quite work well enough. And we tested so many, so many of these solutions. Finally, we said after speaking to everybody from China, chief architects at Adobe, to people who created these solutions, finally we said, well, maybe PDF is not the answer. Maybe we shouldn't just create an accessible PDF.

We actually need to go a little bit deeper. And what if we translate the content of PDF into something that is accessible? So we are looking at different formats. And I was kind of biased because I've been using Markdown for many years.

And I knew that Markdown was going to be answered. But we did look at many different formats. And we settled on Markdown, which is inherently accessible. Most of the UIC, I'm very proud that our team alone has three screen reader native users. And everybody was saying we should just use Markdown.

And that's what ended up what we settled on. So our solution went from turn a PDF into an accessible PDF to translate a PDF to accessible Markdown. And that was a huge unlock for us. Because many, many months and even years, in my case, of trying to find that magic bullet accessible PDF all of a sudden went away.

And then we started to figure out... we started to ask ourselves, how can we do this? Where can we go? And we also had worked a lot on different AI tools at the same time.

I have a background. And working with different large language model technology for about a decade. And I knew that there could be a solution there. But I didn't want to give in to the AI until we absolutely had to.

And we came to that logical part of we can't go... we can't progress programmatically. We're going to need a solution that is an AI solution. That is a deeper solution.

So this is one I handed over to Dylan. Cool. Let's talk about that solution. So you all might be thinking, why Markdown of all things?

I know Blake just hammered on all the different things. But what I like to think about is that Markdown is a democratic file format by design. And by that I mean it's in plain text. You don't need a proprietary software.

You don't need to pay Adobe a couple hundred dollars just to edit your own documents. It's plain text. You can open it up in Notepad, VSCode, anything. It's a plain text document.

It's also... It's also a human readable format without any formatting. So on the right hand side of the page, I have some Markdown examples. So you can see what it looks like from just a non-rendered perspective.

But it's a simple syntax. It's one pound sign equals heading. Two pound sign equals a heading level two. You put square brackets surrounding by a parentheses.

That's a link. You can do dashes for your bullets. It's very simple. Very human.

The other benefit of it is that it's a very semantically rich format. And by that I mean you can convey a lot of meaning by just a small amount of information. You don't need really fancy HTML tags to represent the same things. It's very efficient on how we convey this information.

And what's beautiful about it is that these semantics map directly to HTML. And like we all probably know here, semantic HTML by default is accessible. So if we just use the language as intended, we get the accessibility benefits. No extra work.

And the thing that goes underappreciated I think the most about Markdown is that it's become this lingua franca between AI, humans, and computer programs where humans, like I've been arguing, can read this pretty efficiently. And then when it's rendered, it's beautiful. It's just HTML. But what's great about AI using Markdown is that it's been reinforced to heck on every single Markdown example.

Everything that you see rendered from an AI chatbot is Markdown by default. So they speak it and breathe it. They understand it as deeply as they understand any other language. And the other thing that's excellent about it is that it's really good at being in a computer program. So you can have your...

build systems read in Markdown documents that are then passed into a prompt that then generate a brand new document that can be read by a human. The possibilities are really limitless when it comes to just simple plain text, democratic file conventions, no proprietary companies behind them. So we're going PDF, Markdown, HTML. How do we do it?

And how do we do it at scale? That's the question. And I like to say we do it by using AI as a semantic translator. Blake was mentioning that we can't do it programmatically.

And that's true. It's because there's too many things, too many edge cases. There's too many things that couldn't be tagged effectively. Or there's things that had to be inferred in order to be able to reverse engineer what the intention of an author is to what the markup needs to be.

That's why we need AI acting as this translator between what is visually seen and what is the semantic representation. So we're not just using AI to extract text. We're not using it to just convert into a different file format. We're looking, we're using AI's multimodal, which means that they can have images and text represented in the same space.

We're using its multimodal capabilities in order to bring in the visual language. So then we could reverse engineer what the visual intention is and map it down to the code that is required for the markdown. So we're getting that structure. So we go from that PDF, the visual first print fidelity.

We're using the AI's multimodal reasoning and its coding abilities to go into markdown, which then gives us accessible HTML by default once it's rendered. So that's how our answer is on how to do it at scale. And the AI skeptics in the room, I bet, are just thinking, isn't AI autocomplete? Isn't this all just, you know, statistics at the end of the day?

And it is autocomplete, but there's much more to it than that. So to peel back the curtain a little bit on how AI works, when it is exposed to a lot of information, it's been trained on a ton of information, right? And when it encounters something new, it doesn't just go, oh, okay, this is this sentence in a box. It's actually grouping it together with other, with other nearby concepts.

So if it's reading a news story about Manhattan, it's also conceptually bringing in things about New York State. It's conceptually bringing in things about the subway system. It's all of these different semantic similarity that gets grouped together by meaning. And what's really interesting is when it's grouped by meaning, it goes beyond just language.

So it's not just working in English. It has the same meaning representations in Japanese, in Spanish. It's all talking about the same exact thing. It's just rendering out into a new language.

So what I really hope to do to expand your mind is, what if the language that we're speaking in isn't actually a language that's spoken in words? What if this is a visual language? So how do you say title in visual language? Anyone in the audience?

It's big text centered on the page. And maybe make it the first thing. That's how you say title in visual language. In HTML, you would give that an h1 tag.

You give it the title my paper. Excellent. And then when it's rendered in that HTML, by a screen reader interpreting that semantic tag, that h1 tag, it gets my paper, heading level 1. These are the same thing.

Three different renderings of the same exact concept. This is a new... this is just an abstract language. When I'm saying semantics, it's just an abstract language that represents the title-ness of something.

And if we can represent it in something more inclusive, it gets to the many, many representations. So how do you say link in visual language? Anybody? Blue text underlined.

Excellent. And that in HTML? That is an anchor tag, which then renders it by a screen reader. Visit my website.

Link. Caption?

Italics? Small? Underneath something. Proximity.

That's the key. Proximity, small text, near something. That's establishing a relationship between it. And then how did you do that in HTML?

You do it with the fig caption. Again, it's the same concepts. It's the same underlying language. It's just different representations of that language.

Oops. Not quite. So what's really cool about this is we can have all of these representations. And we can have AI that, again, is that multimodal that can have the image representation alongside the HTML relationship.

Again, mapped in that same conceptual space. That's all well and good, but I'm sure all of us here have experienced AI going off the rails. It's telling us falsehoods. It's not doing something correctly.

It's giving us an incorrect result for remediation. And that's because the AI systems haven't been trained, haven't been reinforcement learned with accessibility in mind. So that's what requires a system. And that system is what we built and we're going to show you today.

So we're happy to share that we have Equalify Reflow. It's an AI harness that will wrap around this process that I'll demonstrate in a little bit where we essentially hold the AI's hand throughout the entire process and then guide it through each step of the way, making sure that it's not making as many mistakes. So before I get to the live example, I'm just going to give a little heads up so you know what to look out for.

How Reflow works is it's through a five-step process. We first extract the document text using a system called Docling developed by IBM. It's excellent for getting the majority of the text out of a PDF. It's also really good at removing two-column layouts from PDFs, which is, if you're a screen reader user, I'm sure brings chills to your spine as well just because you can't control where that focus is going to go in a two-column layout.

The next step that we do is we analyze each PDF. And we're going ahead and we're essentially peeking ahead and alerting the system, hey, this is going to be an academic paper. This is a two-column paper. You're going to be seeing images on pages two and three.

And here's all the other things that you should look out for. And we adjust the prompts ahead of time. That way we're giving that semantic weight to essentially guide and hold the hand of the AI agent so it's able to make the correct decisions at the correct time. The third step of this process is that we extract out and then clean up the heading structure because I'm sure you all know the key to document accessibility is an excellent heading structure.

They're the key to all shortcuts when it comes to assistive devices being able to navigate through. But it's also just what the meaning of the paper needs to cling to. It's the skeleton in which all the content of it needs to be there in order to be full. And then finally, or not finally, but the main show is the translation step.

And we're calling it translation, not just conversion because again, we're doing that visual intention extraction using the model's intelligence and then the coding abilities of it. But what's interesting is that we're going page by page. Each page gets its own agent dedicated to it because I'm sure... if you all have experience with AI agents, if they have too many things going on at one time, they get distracted.

They leave out details. They start hallucinating. So that's one of the key design decisions that we did is keep it focused. One page, one agent.

And because they have all of the metadata of the things to look out for, all of the things that happened previous, like the heading structure, it's able to keep up and it's able to understand where it is in context with relation to one another. And the very final step that we're doing and it's the key to why we call it reflow is that we assemble all of the pages into one single continuous document.

Because pages, frankly, are a thing of the past in my mind. We don't need pages. I want to be able to zoom as much as I want into my document. I want to be able to adjust the text settings in any way that I want.

And having a fixed layout just doesn't work in that case. Some of the other things that we're doing in the assembly step is we're removing things like footnotes and we're moving it to the bottom of the actual document instead of having random footnotes polluting the middle of the content. So these are all the things to look out for. And I'll go ahead and start the demo.

And we'll talk through the demo. So everything will be... Dylan will be the human screen reader. Yeah. See, I'm just doing the mirroring...

or changing it to mirror.

It gets into a range. Oh, yeah.

Extend display. Excellent. Now it's flipped. That's even worse.

Oh, whatever. I'll just do it up here. Do it live.

So here is the EEqualify Reflow view. Zoom in. A little too far. So the very first thing that we're going to do is upload a document.

I'll peek into this document. And we're seeing the Docling Technical Report. This is an academic paper written by IBM detailing the technical specs of the first model that we used for the extraction. I like this example because I really enjoy this cute duck.

And it also tests the system as well when it comes to the AI agents being able to determine whether an image is decorative or informative. Just to say there's a cute duck at the top of the PDF page that Dylan is now scanning into the system. Great. So we've already uploaded the document and then we do the extraction step.

I'll zoom out so we can see a little bit. And just to talk through the user interface, there's basically two columns. One column has the old document and then the second, which is just a PDF document that looks like a standard academic paper. And then the second column is a preview of what this step in reflow has, which Dylan's showing a bit of markdown.

So Dylan, you want to talk about that? Yes. So this is what comes out of Docling and it is a pretty good set of tags. So we're getting all H2s for all of our heading structures, which is fine.

It's not ideal. We're peeking ahead a little bit. That's just a bug in my system.

But essentially, it's just going to be raw headings. It extracts out the text. And there's also some issues with OCR remnants, like an umlaut being detached from the U here. There's also no alt text for any of the images.

So once we finish that step, we go into the analysis step. The analysis step, and on the right-hand side, we have another panel. And it contains all of the metadata that we're identifying ahead of time to, again, alert the AI what is the things that it needs to be looking out for. So we're noting that it's a single column page.

It's an academic paper. And there's images. And then there's tables on pages four and five. And we have the outline.

And we also note other things like there's code blocks available and there's footnotes to look out for. The third step, like we mentioned earlier, is we are figuring out the heading structure. And what's great about this system is that every single decision that gets made by the AI has to be logged. So it's easy for a human reviewer to go ahead and see what's going on.

So we can click into each of the decisions made. And it goes... the outline says Docling Technical Report is heading level one, which is true. But it is currently in heading level two.

And it had 100% match. So it went ahead and auto-applied that. We can go ahead and do that for... It goes ahead and does that for all of them.

And it was able to even detect the initial scan was wrong because of an edge case of the visual H3 and H4 looking very similar. But because of the model being intelligent, it was looking at all of the neighboring heading levels and was able to go, Oh, this should also be an H4 and match the rest of the siblings.

Here's the cool step, the translation step. So we'll look at some of the different decisions that it made. So some things can be very minor, like this very first thing it decided to do was it noted that the word Docling is italicized everywhere. And that isn't translated over.

It doesn't have an impact on the screen reader report. But for the visual report, I mean, the author made it italic. So I think it should be captured as italic regardless of whatever the screen reader is. It should just be a great representation of what the document originally was.

Here's that umlaut that it caught and it corrected. So no issues with multilingual rendering. There is even spelling mistakes that were caught and were corrected. OCR...

Formatting... And Dylan is just going through basically an audit of all the different things our AI tool has done in the translation step. And it's just a big block of different panels that he can open and close. He's opening them to see each different change that happens as he goes over.

Different parts of the audit. He's opening a panel to see more information about what happened before and what it turned to. Yes, thank you. This is a really interesting example where we have a table on page five.

I'll open up the image for... So this image here is a complex table that has many headers within a row. It's a nightmare when it comes to accessibility. But it is also not compatible with Markdown, unfortunately.

But what's beautiful again about Markdown, it renders into HTML natively. So where we don't have support for something in Markdown, we can fall back to HTML. So if we need to represent a complex table, we have the semantics that we can pull from. It's just not as human readable.

Again, the thing that we're striving for is to have everything democratic. I want people to be able to edit their document and feel confident that they can edit the formatting. They don't need to ask permission. They don't need to learn a new language.

It's just, you know, the thing there. But where we fall back to, we can have beautiful HTML that then renders into an actual accessible version of the same complex table. And again, the information is extracted out by Docling, by pulling the actual text out of the document. So we're not having to worry about things like AI hallucinations making up pieces of information about things that are within the document because it was actually just extracted from the document.

And we're asking the AI agent to just simply correct the disparity it's seeing between the visual representation and the semantic representation. And then the final step that we do is the assembly where we go ahead and we move all of the different footnotes to the bottom. We merge it all into one single continuous document. And then I can now scroll freely through here.

Let's see as well. Did I get an alt text for this image? I didn't check that. Dylan is just scrolling through the raw text of the output that we created.

He's seeing all the different raw code looking for an image alt. Yeah, I'll just go ahead and move away. But that is the demo of the system working at working live. So here's a couple other examples.

I'm pulling up just another PDF document. This one's demonstrating the two column layout. This is the standard when it comes to a lot of research papers. And like I mentioned, it is a nightmare for screen readers to be able to traverse a two column layout.

Especially one that isn't particularly well tagged. So on the screen now, we can see this two column page. How to scale code in human dimension. Now all in a single reflowable, single column layout that can be understood by any assistive device and visual user alike.

The next is a course rubric. So the particular thing that I wanted to point out on this rubric that I think is a little bit shaky is the grading policy and breakdown. There's a table here. And part of the table that this teacher published is actually broken.

One of the rows, the total row, is actually broken from the actual table displayed. So this is going to show the model's intelligence to be able to reconstruct what the intended output is and be able to formulate it into the actual markup. So we can scroll down.

Scroll up, right down.

And we can see total is now reconstructed with the actual table. And this is instead of it being a grid view or something like that in Microsoft Word, this is a semantic HTML table that can be really well understood by all AT, which is essential when it comes to what your grade is going to be in a class. And then finally, the last example that I'd like to show you all is what it looks like when it encounters a poster.

So here is a poster which is a nonconventional layout. It's women's rights in challenging times and arts-based civic dialogue. And there are images for everything. The body isn't always obvious of what is the body.

And there's things even down to like the social media handles at the bottom. Kind of hard to read. But when it's ingested into a Equalify Reflow, we can see it extracts out this as a heading level one. The arts-based civic dialogue is heading level one because that was the biggest thing on the page.

Heading level two, women's rights in challenging times. We're getting out that image and we're extracting out all of the event details in semantic markup including all the social media handles. It decided to consolidate them into one single line as well for efficiency.

And then let's see. How do we get this back on to... How does it do the description? That's a great question.

So... Someone asked how does it do descriptions of images? So it does images... descriptions of images in the same way that it's using tools for editing the document.

So whenever it sees a discrepancy for something, it goes oh, I should use the edit text tool to go for the before and the after. It does the same thing when it encounters an image and it actually spawns a new sub-agent whose only task is to write that image description. And I've given it some pretty thorough procedures to follow on identifying whether it's a decorative image or if it's informative. And if it's an informative image, is it a complex image that requires an additional description underneath of it?

So it's just kind of delegating that work out into a sub-agent that gets the necessary context like the page context, the heading structure, the image itself, and the neighboring... text content. And it uses all of that in a little sub-task to assemble that together and then it writes a description and then gives that to the main agent and goes, hey, here's your image description. Use the edit tool.

Finish that for me. And one of the unique things about Reflow that Dylan was just showing you is that we're using lots and lots of specially trained and tuned agents to be able to capture different content. And that was the big unlock of what we're doing. And we're using the word AI harness again and again and again because I like to think of it as when computers came about we made all this software to harness the power of the computer and to do some cool things with all this processing power.

Now with AI we have a lot of different power but if we want to do particular things we use these harnesses or pieces of software we build to focus it and say do this, help with this. And so what we're focused on right now is rolling out Reflow in a way that is responsible but also is going to have lots of different applications. At UIC we're going to have an online service where people can just go to equalify.uic.edu and any user would be able to upload their PDF and change it into a Reflow markdown file.

We're also looking at implementing it with a system called Canvas which is our learning management system at UIC. So if somebody uploads a PDF it can show as Reflow. One of the cool things you can do with Canvas and also WordPress is what we've also been building with is we can actually convert a PDF into a Canvas page. So we don't just have to convert it into Markdown.

With Markdown you can convert it into anything. So a lot of professors we've heard from have told us we just like PDFs. We don't even like Canvas. And so Reflow actually feeds two birds with one scone where they can just upload a PDF it can create their Canvas page they don't have to deal with Canvas and that Canvas page is accessible.

So that's what we're working on with that. And with WordPress similarly when you have a post Reflow can create posts out of PDFs and it can also add a little bug near any PDF. link that a screen reader user can click on and immediately get the accessible version of Reflow. So just quickly of the economics around Reflow.

Again we were looking at five to ten dollars a page. Over a million dollars annually to do remediation. Like today and we are not using the most cost efficient models right now. We're looking at around ten cents a page.

This is actually actively going down. As Dylan was just telling me about another model that came out today. So it's actively going down and what we hope to do is have this truly open source and be able to use open source models within it so that cost is theoretically the cost of buying a computer. And so right now just today we have about a 50x reduction.

We are starting to implement this. With this title tool rule one of our big plans is to start scanning the products that work within Reflow. So again, Reflow is open source. It's licensed through AGPL3 version 3 for those of you who know that basically it's open source into the cloud.

Pretty radically open source. We're rolling it out at UIC as we speak. Then we're rolling it out to different partners. And then we'll roll it out to the public.

This is hopefully going to be a super quick pass. But within that partnership anybody who wants to start playing around with it can go to equalify .uic.edu slash reflow. So that's equalify.uic.edu slash reflow. And you can just sign up a basic form there.

We're rolling it out to everybody who signs it up in a similar way. And if you are interested in getting deeper in touch with us, you can always contact us. My email is very simple. It's the number 3b at uic.edu And you can contact us about that.

But Dylan, you have one last word in the last minute. Thank you all for having us. Thank you all.

Thank you.

Sorry, I'm getting a vert cold. Why not? Testing. There we go.

Hello. Dylan and Blake, thank you very much. Really cool stuff. I'm... really excited to start playing around with it as soon as I get a chance.

We're going to have about 8 minutes for Q&A. And I'll go back and forth between questions coming through online and questions coming from the room here. So I'll start in the room. Hang on.

Sorry, I'm getting a vert cold. Hey Dylan and Blake, thanks so much for the incredible hard work. I imagine this was quite the endeavor. So I love the fact that it's apparently an agent-to-agent framework, at least what Gemini and ChatGPT both concluded.

The question that I have for you is... Since you said it's open source, can people incorporate other agents to deal with... Like you pointed out with the two-column corporate documents that tend to have an executive summary because it's designed for a certain type of audience or things with heavy jargon. Can people try to contribute to this by making their own agents for those edge cases? And if so, how can that work?

I'll jump in real quick. Why not? Yes. So we have trained and are working on academic papers right now because that's UIC's biggest number of issues where academic papers, student organization posters, and syllabuses.

Those are like where we're focused on. But we want this to be a universal project and so the AI harness system that we built is a system that is supposed to be expanded and we do hope that technically folks do contribute new agents and new ideas and that is one of the ways, one of the reasons that we're open source because we want this to be a platform that people use to build new agents for their very niche problems.

So when those problems come up from someone else, anywhere around the world that they can use them and solve them. Cool. Question online. Have you already tested it with math formula components?

In PDF? And does it translate successfully into markdown and then to HTML? Not yet. That's in our next roadmap.

That's one of the hardest edge cases and it's one of those unsolved problems right now. But we're seeing a lot of like a lot of hopeful outlooks when it comes to using MathML and stuff like that. The agents understand that quite well. It's just getting the prompts ready for the translation to prep it.

Sorry about my voice. Cool. I've got... in the room.

Yeah, it's funny about the math. As I'm sure you've discovered the process of turning PDF into something that you can read depends a lot on how the PDF was made. So like for the first one, that's was clearly made using LaTeX or TeX.

And so your process is sort of just reverse engineering the process of making the PDF. And if you want to do the math all you have to do is look up the arXiv, get the TeX, take that and do a conversion to HTML and it's all no AI. It works really great. So the most impressive one is your poster.

I agree. And even that the second one those are pretty impressive things. But can you comment on on what do you need different models depending on how the PDF was produced? We don't need different models depending on the PDF that's produced because we're utilizing first a pass with docling that's using a combination of tiny OCR and extracting out the document semantics.

So it's about as good as the OCR output that we're getting from that first extraction step. And then the next step is that AI harness that's asking it to look at the visual affordance, the visual presentation of it versus the textual of it. And it's it doesn't matter what because we're just uploading an image to the AI model. It doesn't matter if it was born native or if it was born digital native or if it were handwritten even.

So does your process look at the PDF code or does it render the PDF code all to a raster image and then put the raster image through some uniform process? Our system uses the rasterized image mainly. The docling step, which is the first text extraction, uses the PDF document semantics. So it's actually using the PDF text.

It does both. So even before we had AI involved, I was trying to do the docling step and building a tool that did the docling step. The tool called iText to kind of look through the PDF and parse it. And it could never get to the... we needed to do both.

And the other reason we do both is because it's a lot less computational heavy to look at the PDF code first and make large summary decisions based off of that instead of having to run it through larger and larger models to do that. It's cheaper too. Online question.

How does the workflow handle personal or confidential information security? Great question. We've thought of that. So we are using an open source project from Microsoft called Presidio.

And it's actually like it's like regex on steroids. It comes up with all these different conditions in which it can bring out things as broad as first and last name to as narrow as social security numbers. And any document that gets uploaded through the system outside of the demo obviously has to go through that Presidio step. And it flags all the things that it's identified as personally identifiable information.

And then makes the faculty member confirm that this is intended for publishing to a cloud model that it's safe to do so. The other piece of it that I would say is a ring for open source, especially in the AI world, is we're not saying use the OpenAI model. We are building a system that is a harness for any model and also any environment that you're running the model. So you could theoretically have a completely closed source system that you're running the entire system without exposing it to the cloud at all.

Cool. One from the other room.

First of all, thank you. That was an amazing presentation. So my question is, I'm from the blind low visually impaired community and this is what's going to help us. Is the website accessible to screen readers and can the website be made like high contrast for those who have some vision but can only see in high contrast?

Yeah, so there's a few aspects to that question. First of all, the website that the quali.uic.edu slash reflow is accessible, that's a marketing website. The tool itself, what Dylan was demoing and walking through is more of an example tool. The actual need of it is a service.

So it's an API, which means it's not just a website that people would go to, it's something that people can build websites from. But Dylan, I'll let you talk about the actual demo website that you have. The demo website is also quite accessible. I'm using a lot of semantic elements and I've also added in a lot of keyboard shortcuts and skip links that go to all of the different places on the website that you would care about. So if you wanted to go and look through all the different steps that we were discussing, or if you wanted to jump straight to the markup, or if you wanted to go to where all the decisions are made, they're all skip links away.

One of the things that we really like to do related to this is make this accessible tech. You can embed it in a screen reader, because whenever a screen reader comes across a PDF, it should ask you, do you want to convert it to Markdown and listen to it. At the very least, within a few months, we've been talking to some folks at GitHub about this. We really want to do a hackathon around just making assistive technology around this.

So at the very least, it could be a Chrome extension that whenever you came to a PDF, you could convert it. Yeah.

I feel like we could continue Q&A for another 45 minutes, because there's so much good stuff to dig into here. We've got about five minutes left before we need to let our captioner and ASL interpreter go. So maybe, does anybody feel like they've got a, this is always risky, anybody feel like they've got a one minute or less question that can be answered? Otherwise, we can continue the conversation later.

And for anybody online that didn't get your question answered, I apologize. But definitely connect with these guys and continue the conversation with them async. b3b.uic.edu b3b.uic.edu. Alright, hang on.

We'll take one more quick one, hopefully. And then we'll do our wrap up. Okay, if you were to start this project from scratch, would you do anything different? Hmm.

Yeah, I've redesigned this project maybe about three or four times. And I have another one that I started, and then I decided I needed to just polish this one. But yes, there's something called graph view, where I would solve it as a graph problem. And it's probably too technical to go into for a one minute question, but yes, I would. But all of us...

Software's never done. No. Blake, Dylan, thank you guys both very much. Quick round of applause.

Really appreciate it.