Add mutex to limit the number of goroutine to lookup meta table

[YutSean]

In our production environment, when a table contains a large quantity of regions (more than 1000 regions), the client will send too many scan requests to meta regionserver when the local cache is invalid. This results in a rapid heavy pressure to the meta regionserver and the meta regionserver may be dead. To protected the meta regionserver, add a mutex to limit the lookup of meta table will improve the stability of the whole cluster.

[codecov-commenter]

Codecov Report

Merging #164 (d154d71) into master (f1ffe9f) will decrease coverage by 1.62%.
The diff coverage is 12.69%.

❗ Current head d154d71 differs from pull request most recent head b447f9d. Consider uploading reports for the commit b447f9d to get more accurate results

@@            Coverage Diff             @@
##           master     #164      +/-   ##
==========================================
- Coverage   70.68%   69.05%   -1.63%     
==========================================
  Files          25       27       +2     
  Lines        2180     2243      +63     
==========================================
+ Hits         1541     1549       +8     
- Misses        537      588      +51     
- Partials      102      106       +4     

Impacted Files	Coverage Δ
client.go	64.77% <ø> (ø)
hrpc/balancer.go	0.00% <0.00%> (ø)
hrpc/move.go	0.00% <0.00%> (ø)
admin_client.go	54.74% <42.85%> (-1.36%)	⬇️
rpc.go	81.25% <100.00%> (+0.12%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3489911...b447f9d. Read the comment docs.

[tsuna]

This limits the concurrency to 1 though, maybe we should use a semaphore to allow multiple concurrent lookups to proceed? Do you have any metrics to share in terms of the performance impact this change has?

[YutSean]

I checked the HBase community Java Client implementation. It just use a lock to ensure only one thread check meta info at one time and in our usage, only one goroutine to check meta info is not noticeable by the users. Btw, meta lookup is a very fast process, use mutex can protect the regionserver where the meta region located.