Tidal disruption events (TDEs) are observationally classified into two classes – optical TDEs mainly radiate in the optical/UV bands while X-ray TDEs mainly radiate in X-rays. The unified model of TDEs proposes that these optical and X-ray TDEs are intrinsically the same, and the different TDE types are caused by different view angles with respect to optically thick outflows. Therefore, this model predicts that optical and X-ray TDEs should reside in similar galaxies. We investigate this prediction by measuring the star-formation histories (SFHs) of TDE host galaxies for representative samples of optical and X-ray TDEs. We compiled their multi-wavelength data and fit their UV-to-IR (infrared) spectral energy distributions (SEDs). Our SED fitting is based on Prospector, which provides highly flexible, nonparametric SFH models to reliably constrain the recent SFHs of our sources. We found that the typical resulting SFH of optical TDE hosts is consistent with post-starburst ones with recent (< 1 Gyr) rapid, strong quenching, but SFHs of X-ray TDE hosts tend to show less strong-quenching features. This may indicate that more physics linking the TDE type and TDE hosts, which are not considered in the TDE unified model, are still in a veil, and we briefly discuss more scientific implications of our results.