-cd can be avoided, inspecting the database via phpMyAdmin or programs like SequelPro and add the new column manually. After the manual "migration" you need to update versions table to the new database version.

UPDATE versions SET val = X

Funny side fact: versions, together with gymdetails are the the only tables we use a plural wording. Especially for versions it's cool, because it only holds one single entry.

U can add cp to webhook too ?

Could you please add cp_mutliplier to check Pokemonlevel?
I can give you JS code for calculation but adding the column would be perfect. Thanks!

Can we dont use csv files for the 25 and 30 level accounts? How should the format be then? I dont see any information about that in the encounters.md file.

@tokar86a This PR is for development, not customer support. Be patient until things are finalized.

Why do we specify pokemon to encounter for IV/CP by their pokemon IDs in iv-whitelist-file and cp-whitelist-file, while we have to specify them by their name in webhook-whitelist-file ? Wouldn't it be more consistent to chose for one or the other ? I also guess lots of people have no reason to send pokemons for which they don't scan IV to their webhook so having similar config file would allow to maintain only one. I personally do prefer pokemon IDs as lots of maps are localized and Pokemon IDs are the same for everyone while names are not...

@sebastienvercammen Maybe if I give you an example you would understand why locking in next doesn't help that much.

Problem 1:

1.- Account A scans point 1 get 1 mon and need a IV then calls next() and get account X for it
2.- While in the same thread X calls for log in, this makes a request, sleep and changes threads
3.- Account B was waiting for point 2 results (point 1 and point 2 are alongside each other) parses results and needs to encounter 1 mon for IV then call next() and could possibly get account X again (thread safety is only inside that function, speed limit is ok as it is near, etc)
4.- While in thread B we call another login on X this could lead to context switch
In any case A encounter will fail due to login auth token will be invalid.

While complying with the speed limit if there is thread starvation due to high cpu usage it could even happen with points far away.

Problem 2 (only happens if context switch could happen anytime):

1.- Account A scans point 1 get 1 mon and need a IV then calls next() and get account X for it
2.- Before login there is a context switch.
3.- Some time passes (maybe a couple of minutes)
3.- Account B was waiting for point 2 results parses results and needs to encounter 1 mon for IV then call next() and could possibly get account X again (thread safety is only inside that function, speed limit is ok as it is near enough, etc)
4.- B login, encounter, etc.
5.- Thread A gets priority again tries to login, no issues but we are violating speed limit as we are traveling from B to A without any control.

Problem 3:
1.- Account A scans point 1 gets 3 mons that need encounter
2.- We call next() 3 times and log in 3 times with the same or with different accounts.

Cpu starvation is not that strange as python threading is bounded to 1 core and we are using hundreds or threads in any instance, I have seen situations where my db queue skyrockets just because the db threads doesn't get priority

@friscoMad That makes more sense. Fixed in next commit.

Problem 3 however would use 3 separate accounts for the scans (assuming we're not entering Problem #1), as intended. The login rate is a valid point, and one that I started a discussion about on Discord (login rate vs memory usage).

All of the threads that our legacy code uses are actually completely unnecessary. They're bad practice, slow down the process (have you seen the number of threads we use...?!) and lead to unexpected behavior. Luckily they're already put on the roadmap to be reworked.

@sebastienvercammen
For Problem 3 I was thinking that using the same account for all encounters is the best option using the step location you can see all the mons so you can encounter them, there is no need to use 3 different accounts, the problem in that case is that you need to check the highest level needed to encounter all of them

@friscoMad It doesn't have to be the same account per se. If they're truly so close to one another, it will have the same result but it would spread encounters over different accounts. The problem lies in the login rate, which would be solved by storing the API object, but as mentioned before, that's being discussed.

@sebastienvercammen I don't the reason why not using the same account for everything as it was done previously for all mons found in a scan using the scanning account. If the account from the pool can reach the step it can encounter all in 1 login without any violation (and we can use again the encounter delay).

pseudo code (missing a lot of checks) to explain myself:

use_cp = false
use_iv = false
encounter_mons = set()
for mon in pokes:
  if mon in IV_white:
    encounter_mons.add(mon)
    use_iv = true
  else if mon in CP_white:
    encounter_mon.add(mon)
    use_cp = true

acc
if use_cp:
  acc = next(30, location)
else if use_iv:
  acc= next(25, location) or next(30, location)

login(acc)
for mon in encounter_mons:
  delay()
  encounter(acc, mon)

@friscoMad Remember you also need to consider the distance between each of the encounter_mons. It should be trivial, but needs to be checked.

In an upcoming commit, I've added a toggle to enable storing the API object to re-use it. So the host can choose to increase the login rate, or to increase memory usage.

I'll do some testing on the distances and re-using a single account for all encounters. If I can confirm our thoughts, I'll look at adding it in.

@sebastienvercammen Confirmed your latest commit does solve my issue with account stopping after 1 encounter. Thanks

All of the previously mentioned points are now committed.

Why so much trouble having different files and all when a simple check to get_player_level response could determine account level?
CP being included when information is sent to front-end and to the database is one thing, having separate account managers/queues, separate csv files etc. should be another thing.
I feel we really need to put accounts in database, this would drastically improve how easy we can pick an account with a specific level or close to a location instead of having accounts tied to instances.
This is a good starting point but it's a bit "clunky"... for instance, I manage 44 instance so ultimately I would need 44*3=132 csv files with accounts to enable IV scanning in all my instances which seems impractical. Also I don't know exactly how you and Seb did the lookup on the "high-level.csv" but you can't/shouldn't use the same file for all instance or accounts will be shared between instances. I saw that seb uses 'in_use' flag for each account but it's kinda silly because another instance could have picked that account as well and since 'in_use' is not a state shared between instances -> kaboom two logins/scans in parallel in different locations.

@neskk this is already addressed by some work on my fork. I thought exactly like you just brought up. Just, this is a first implementation to keep the map up to date with what Niantic pushs into the scanners way. Improvement is coming step by step.

@neskk Because the implementation we decided to go for is one that reduces the amount of integration required with legacy code. This implementation is simple and very easy to separate. This includes not reworking the legacy code to first send an additional request just for its level.

A proper rework includes reworking account handling, and ultimately scan handling. This has been talked about several times, including in this thread.

High level account files shouldn't be shared by instances, as has always been the case for account files for the past 10+ months, and I'll only repeat this one last time: we need to rework account handling to allow account file sharing and to properly distribute accounts. But that's outside of the scope of this PR.

in_use isn't a flag to share between instances, it's to avoid one account being used in two separate threads because of thread starvation.

And it would be 88 rather than 132 files with your 44 instances, because we only have two account files: one for accounts to scan with, one for accounts to encounter with.

@sebastienvercammen upon login (first_login), get_player should always be requested to check warning flags, check if account is banned, check tutorial state (remove necessity for -tut flag) and perhaps try to collect level up rewards (something that current -tut with pokestop spinning doesn't do). These are all things easy to implement, I did it in less than an hour and one extra request upon first_login isn't relevant in the overall scheme of things.

@neskk It is relevant when it only adds more code to the list of code that needs to be scrapped, because you'd be reimplementing the same duplicate code yet once again, adding one more place to get rid of it.

And if you implement it with an abstraction, you might create an abstraction that fits for your little rework but not for the proper account handling/scanning rework.

I said I wouldn't repeat myself again but I indirectly have so I'm leaving that discussion.

@sebastienvercammen impressive how you can write so much and tell so little.
Duplicate code? Just remove get_player_level() (account.py) and instead redo the method to parse player data into the account dict: account['level'] -> parse_map() already receives account dict as param, so it's an improvement to remove get_player_level() there because you don't have to parse player_level in each parse_map() call as it does now.
This new method "get_player_state()" would be called only upon first login to save player level, tutorials, warnings, etc for later because it can be useful in several stuff including CP/IV scanning.

Niantic seems to have changed something recently that might affect this PR. https://www.reddit.com/r/TheSilphRoad/comments/67ypx2/different_stats_and_moves_between_level_26_and/
https://www.reddit.com/r/TheSilphRoad/comments/67yuzc/it_seems_pokemon_spawn_stats_may_now_be_entirely/

The resent changes make this pr more simple, just only lvl 30+ acc in the pool ?

Adjust title ;-)