We do use these terms differently for males and females. No, let me be more precise right from the start, as I am hoping to make some distinctions. We use these terms in almost the same way in some situations, while using them differently in others when referring to males and females of different ages and status. This is generally to the disadvantage of females and their perceived status, but there are some reverses.
First, people of both sexes use the term for themselves for single-sex groupings that are primarily social. Going out with the boys in your 80's is not different in tone from your sister going out with the girls in her 80's. This is especially true if these are age-mates one actually has known for years. The term signals informality, but not the least hint of disrespect.
There is a slight weakening when one uses it about others. A husband who goes out with The Boys might reference that his wife is out with The Girls at the moment and vice-versa if each uses that term themselves. But if not, there might be a hint of superiority. That flows both ways about equally, I think. This is even more true of children speaking about their parents. It is possible that there is not the least amusement or condescension when talking about Mother going out with The Girls, but more likely, there's at least a touch of it. It is very likely to be present in the comment of a granddaughter talking about her grandfather going out with The Boys.
Yet even here some other factors have quietly entered in. The age-reversal of speaking of people 40-60 years older than you as boys or girls is humorous in itself. Children don't use the words that way in other contexts. They lack the abstraction and metaphor to even understand that the reference to boys is subtly humorous to a 70-year-old and is signalling affection. This collision of meanings does not go away quickly in children, even into their 20's before they are quite comfortable with it.
This is a good spot to insert that there are going to be regional and cultural complications here. In New England people are very likely to use the term "guys," and there is no good female equivalent. "Gals" is more southern and midwestern, though we do use it some, especially in combination with guys.
At the other end are the clearly sexist references, in contexts where 25 y/o males are called "men" but 45 y/o females "girls." There aren't so many of these now, but they persist. The gap has narrowed, anyway. The complication here is that age is not the only factor directing our choices. The status of the person or group of people is usually an even stronger driver of whether they get called man or boy, woman or girl. This starts to curve back upon itself. For most of history men have had the higher status jobs. Do we ascribe greater status to them in our reference because of their maleness, their age, or their status? Cart, horse. (I would very much like to digress here and discuss what preferable metaphor I should use instead of cart/horse when there are three choices. But I refrain. Though it bothers me, and it's a wrench to leave it behind.)
Racial discussions had a steady undercurrent in the 20th C of not using "boy" or "girl" for any grown African-American, hoping that language change would bring social change. This turned into a huge factor in original feminism, as there was suddenly pressure to use clumsy terms like chairperson. The consensus at the time was that such language changes are not effective when imposed by fiat, they have to evolve naturally. I don't think that has turned out to be entirely true. I think the enforced language changes have had mixed effect, but have mostly worked in the direction their advocates wanted. There has been a cost in resentment and less-graceful, less-comfortable language, but I think the changes have indirectly changed thinking. I don't hold with that ever-popular linguistic myth that changes in language create changes in thinking, but I do believe that the artificial awareness of language has reminded people of the potentially insulting nature of some traditional phrasings. Christmas carols and other hymns are made worse, and not only because we are cutting ourselves off from our own ancestors. But not as worse as was predicted, and I believe the gains elsewhere in equality have compensated.
Though again: Cart. Horse. Perhaps the changes were going to happen anyway and all the language battles have been irritating to no effect.
In between all this is a swamp of usages that depend on competing forces and contexts. One of my patients years ago was a black man my own age. He had been an honor student at a rival highschool and I knew friends of his from a summer studies program at St. Paul's. He was bipolar with some additional, but not severe alcohol problems. He would slow down pretty quickly on medications and we would have a fine time talking. He only had 3-4 admissions in a 2 year period and then I never saw him again. I liked to thank him for going out for walks with me so that I could look like I was working when I was just hanging out with a guy. But when I was talking with his mother on the phone, I completely lost the context that she did not my age or my connection, and when I said "He's a very smart boy" there was a chill on the other end of the line. My error, not hers, though I have said the same about other men my own age.
There are male-female contexts that are equally perilous. I have read a few times of women quietly shuddering the first time a young man called them "ma'am," and developed the habit decades ago of calling every woman "miss." Thank you, miss. Not every woman comments, but when they do the response has been overwhelmingly positive, usually humorous. Yet not always. A very few times I have felt instant resentment to my word-choice. I understand it. You would think the resentment would lessen as I aged, as a large percentage of females are indeed young to me now. But I think the offense is increasing. I don't know if that is a culture change, where my use of "miss" is heard as an equivalent of "girl," or some assumption that because I am an old guy I must be a benighted individual who still adheres to ancient attitudes.