How inclusive are our classrooms really?

Look carefully at this picture. Is this the sign that is implicitly posted in our classrooms?

I thought that my teaching style was pretty inclusive for ESL students - I try to avoid or at least explain English idioms, I make a lot of eye contact and never speak to the board, I provide written as well as verbal instructions, I try to include cross-cultural examples, I ask students to work in groups so they can practice explaining concepts to one another...

But this past week I received a message from a student who felt that she had not done well on the first exam because her native language is Japanese, and it takes her longer to read and answer the multiple choice and short answer questions I required her to complete. I had prepared what I thought was a reasonable exam for a 45-minute time period - for an English speaker. It hadn't occurred to me how difficult my exam would be if you had to first break the language barrier before being able to really engage with the material.

I didn't know how to respond. Speaking English as a second language is not a disability, and therefore is not covered by the ADA academic accommodations policy. Generally my own policy is that unless a student has a documented disability, I don't provide accommodations that I am not willing to extend to all students in the class. I can't give everyone extra time (even if I wanted to, there's another class in the same room immediately before and after mine), and I can't make the exam much shorter without sacrificing the validity of the assessment tool (I wouldn't be able to adequately sample the material we had covered).

Unfortunately it seems that throughout higher education, unless they are taking an ESL course (such as those offered by our Department of Linguistics), ESL students have to "sink or swim". Is this the way it should be, or is there a way for me to provide ESL students with accommodations without native English speakers feeling that they have been treated unfairly? What is the equitable solution here?

I'm eager to hear your thoughts...