Add VisitAllEntries method to iterate all entries in cache. by andreiavrammsd · Pull Request #3 · VictoriaMetrics/fastcache · GitHub

Iterate all entries in cache with a given callback with signature f func(k, v []byte) error. Iteration stops if error is returned from callback.

Outdated:

Retrieves cached keys by regex pattern.

type User struct {
	ID       int    `json:"id"`
	Username string `json:"username"`
}

const userCacheKey = "user:%d"

cache := fastcache.New(1000)

// Insert data into cache
u := &User{ID: 1, Username: "John"}
b, _ := json.Marshal(u)
cache.Set([]byte(fmt.Sprintf(userCacheKey, u.ID)), b)

u = &User{ID: 2, Username: "Doe"}
b, _ = json.Marshal(u)
cache.Set([]byte(fmt.Sprintf(userCacheKey, u.ID)), b)

// Retrieve all users
users, err := cache.Keys(`user:\d+`)
if err != nil {
	panic(err)
}

fmt.Printf("Found %d user(s)\n", len(users))

for _, u := range users {
	user := &User{}
	if err := json.Unmarshal(cache.Get(nil, u), user); err != nil {
		fmt.Printf("Error retrieving key \"%s\"", u)
		continue
	}
	fmt.Printf("%#v\n", user)
}

Codecov Report

Merging #3 into master will increase coverage by 1.14%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master      #3      +/-   ##
=========================================
+ Coverage   76.05%   77.2%   +1.14%     
=========================================
  Files           4       4              
  Lines         497     522      +25     
=========================================
+ Hits          378     403      +25     
  Misses         65      65              
  Partials       54      54

Impacted Files	Coverage Δ
fastcache.go	89.79% <100%> (+1.49%)	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78aabb6...39cf0ef. Read the comment docs.

Will it be better to filter keys outside fastcache?
Just get all the keys, filter them, fetch matched key-value pairs.

Will it be better to filter keys outside fastcache?
Just get all the keys, filter them, fetch matched key-value pairs.

This won't be better, since returned keys for huge number of cached items (typical workload for fastcache) would require a lot of additional memory and pointers.

Currently it is called under per-bucket read lock. This means the bucket contents cannot be modified until all its entries are visited. This may take a lot of time for slow callbacks. I'm unsure whether this is OK.

This also would lead to deadlock on an attempt to call Cache.Set(key, ...) inside f.

If we mention these possible situations in the docs, do you think users will still ignore them and they will end up in undesired states?

If we mention these possible situations in the docs, do you think users will still ignore them and they will end up in undesired states?

I'm absolutely sure :) There are the following choices:

to document all the restrictions and caveats in the hope users read and adhere the docs
to harden the implementation against users who don't read docs. I.e. do not hold the lock during f call and copy key, value args passed to f. This would slow down the implementation
to fork fastcache and implement this in the fork

I'm leaning to the last solution. Let's leave the PR open for a while.

Could we push on a queue all operations (get, set, del...) while VisitAllEntries is working? And when it finishes, we run all the operations (in the same order, of course).

Retrieves cached keys by regex pattern.

Please don't put everything inside doc only. You can write the limitations, caveats and few basic complete examples on the readme.md

Things to look out for should be most visible.

There are a lot of nice features and TTL is definitely needed here.

Please do make TTL ready at least as soon as possible.

I think the format of method can be implemented and set,
the function / method / strategy / optimization can be improved on later.

e.g. SetWithTTL(k,v,t) etc or just SetT(k,v,t) etc can be implemented already.
As for how the actual method should be, we can improve on overall later.

No one will read the doc complete but at least they will definitely read the readme.md AND a wiki section will be nice with FAQs section

some case TTL is must and cool, but some others cache had.
but , TTL is no need for high performance cache, there another ( easy ) way to update cache item or remove it.

fastcache is perfect in my trial project and running in product now.

@andreiavrammsd do you need any help to finish this PR? fastcache is perfect for my use case, it's missing just the feature to iterate over all the entries.

@diegobernardes, the PR is finished but not accepted because of performance concerns. Please see the discussions.

Is there a feature for VisitAllEntries()? How can I code this / add to personal repo?

Why not develop a function of simple Keys without regex pattern，I think there will be fewer performance problems 😄

valyala requested changes Dec 4, 2018

View reviewed changes

Comment thread fastcache.go Outdated

Comment thread fastcache.go Outdated

Comment thread fastcache.go Outdated

andreiavrammsd force-pushed the keys branch from d09c099 to 932eb55 Compare December 5, 2018 21:23

cristaloleg reviewed Dec 6, 2018

View reviewed changes

Comment thread fastcache.go Outdated

valyala reviewed Dec 6, 2018

View reviewed changes

andreiavrammsd force-pushed the keys branch from 932eb55 to dec9328 Compare December 6, 2018 17:47

andreiavrammsd changed the title Add Keys method. Add VisitAllEntries method to iterate all entries in cache. Dec 6, 2018

andreiavrammsd added 9 commits February 10, 2019 16:32

Add Keys method.

c9ef97c

Retrieves cached keys by regex pattern.

Test with invalid regex.

af06015

Implement cache items visitor.

591074d

Improve docs.

6456ba3

Improve tests.

2593d33

Move visitor function to bucket.

0790c44

Limit key and value capacities.

98d9aa3

Skip overwritten entries.

7f5248e

Improve docs.

39cf0ef

andreiavrammsd force-pushed the keys branch from dec9328 to 39cf0ef Compare February 10, 2019 14:36

valyala mentioned this pull request May 29, 2019

Can you add a way to iterate all values in memory? #9

Open

valyala mentioned this pull request Jul 29, 2020

Cache expiration recommendations #33

Open

cristaloleg mentioned this pull request Jan 22, 2021

Iterator support? #43

Open

ehnuje mentioned this pull request Jan 31, 2021

Support Iterator for fastcache #44

Open

cristaloleg mentioned this pull request Jun 22, 2021

can you support a method for iterator all keys in fastcache #53

Open

+              			idx += keyLen
+              			value := chunk[idx : idx+valLen]
+              			if err := f(key, value); err != nil {

Add VisitAllEntries method to iterate all entries in cache. by andreiavrammsd · Pull Request #3 · VictoriaMetrics/fastcache · GitHub

Uh oh!

Add VisitAllEntries method to iterate all entries in cache. - #3

Add VisitAllEntries method to iterate all entries in cache.#3andreiavrammsd wants to merge 9 commits intoVictoriaMetrics:masterVictoriaMetrics/fastcache:masterfrom andreiavrammsd:keysandreiavrammsd/fastcache:keysCopy head branch name to clipboard

Conversation

andreiavrammsd commented Dec 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Dec 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cristaloleg commented Dec 4, 2018

Uh oh!

valyala commented Dec 4, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

valyala Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

valyala Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

andreiavrammsd Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

valyala Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

andreiavrammsd Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hiqinternational commented Jun 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tsingson commented Aug 20, 2019

Uh oh!

diegobernardes commented Nov 4, 2019

Uh oh!

andreiavrammsd commented Nov 4, 2019

Uh oh!

hiqsociety commented Mar 12, 2021

Uh oh!

miaoyc666 commented Nov 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Add VisitAllEntries method to iterate all entries in cache.#3
andreiavrammsd wants to merge 9 commits into
VictoriaMetrics:masterfrom
andreiavrammsd:keys

andreiavrammsd commented Dec 3, 2018 •
edited
Loading

codecov-io commented Dec 3, 2018 •
edited
Loading

hiqinternational commented Jun 5, 2019 •
edited
Loading