There are some wonderful dirt road rides in the Tucson area, all of which can be done on a gravel bike. However, many of the best ones involve driving to the start of the ride.
One thing to note if you are doing any of the rides on the north side of Mount Lemmon is that dirt roads are occasionally paved. For example, between the first and last times I rode up the Control Road from Oracle, another 1.6 miles was paved. So be aware that my descriptions of where dirt roads end and paved roads start may be slightly off.