add files to compute basic stats on pseudo crawl dataset by SaulLu · Pull Request #382 · bigscience-workshop/data_tooling · GitHub

As discussed with @thomasw21, this PR add basic slurm and python scripts to compute an intermiadiary metadata dataset and some statistics for the Pseudo Crawl dataset

for more information, see https://pre-commit.ci

LGTM! A few comments but should be good.

We never us it. Let's remove it.

In fact, it seems to me that I used use_datasets_caching

I'll removed save-batch-size

Suggested change

      
                    "--save-path-stats-full-json", type=str, help="Where to save the stats json."
          
                    "--save-path-stats-full-json", type=str, required=True, help="Where to save the stats json."

Suggested change

      
                logger.info(f" --- Statistics not already computed for seed id {args.seed_id} ")
          
                if not args.use_datasets_caching:
          
                    datasets.set_caching_enabled(False)
          
                else:
          
                    logger.info(
          
                        f"the datasets results will be cached at {config.HF_DATASETS_CACHE}."
          
                    )
          
                logger.info(f" --- Statistics not already computed for seed id {args.seed_id} ")

In fact, it seems to me that I used use_datasets_caching

Suggested change

      
                    batched=False,
          
                    batched=True,

You just have to code a batched version of get_length_text

Yeah I'd recomment os.rename. subprocess doesn't tell you if it fails.

Suggested change

      
                save_path = Path(args.save_path_stats_full_json)
          
                tmp_file_name = f"tmp-{str(save_path.name)}"
          
                save_path_tmp = os.path.join(save_path.parent, tmp_file_name)
          
                logger.info(f"Saving the dataset at {save_path_tmp}")
          
                ds_html.to_json(
          
                    save_path_tmp,
          
                    batch_size=args.save_batch_size,
          
                    num_proc=args.num_proc,
          
                    compression="gzip",
          
                )
          
                logger.info(f"Moving the saved dataset to {str(save_path.absolute())}")
          
                save_path_full = Path(args.save_path_stats_full_json)
          
                save_path_full_tmp = save_path_full.rename(f"{save_path_full.name}.tmp")
          
                logger.info(f"Saving the dataset at {save_path_full_tmp}")
          
                ds_html.to_json(
          
                    save_path_full_tmp,
          
                    batch_size=args.save_batch_size,
          
                    num_proc=args.num_proc,
          
                    compression="gzip",
          
                )
          
                logger.info(f"Moving the saved dataset to {str(save_path_full.absolute())}")

two things:

move with os.rename, or shutil.move
You're not scared of overiting previous full_json?

You need to add the new seeds. And I need to update the csv with the new seeds ....

@SaulLu can you edit and merge? Thanks!

Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>

for more information, see https://pre-commit.ci

Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>

@HugoLaurencon, do you need it urgently ?

@SaulLu No, no worries! It was just to tag you in case you didn't see the comments from Thomas

SaulLu added 3 commits February 1, 2022 16:12

add files to compute basic stats on pseudo crawl dataset

9368b49

Merge remote-tracking branch 'upstream/master' into compute-statistics

c388e47

update statistics

19c004e

SaulLu force-pushed the compute-statistics branch from 96fcc41 to 19c004e Compare February 2, 2022 08:25

[pre-commit.ci] auto fixes from pre-commit.com hooks

11d0fd1

for more information, see https://pre-commit.ci

HugoLaurencon requested a review from thomasw21 February 4, 2022 17:12

thomasw21 approved these changes Feb 4, 2022

View reviewed changes

SaulLu and others added 4 commits February 8, 2022 15:33

Update dashboard/python_scripts/compute_stats.py

d4f38a5

Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>

Update dashboard/python_scripts/compute_stats.py

f644aca

Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

81f0062

for more information, see https://pre-commit.ci

Update dashboard/python_scripts/compute_stats.py

e17c42d

Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>

+                  with open(save_path_tmp, "w", encoding="utf-8") as f:
+                      json.dump(data_stats, f, ensure_ascii=False, indent=4)
+                  logger.info(f"Moving the saved dataset to {str(save_path.absolute())}")
+                  subprocess.run(["mv", save_path_tmp, str(save_path.absolute())])

+              #SBATCH --partition=cpu_p1
+              #SBATCH --time 10:00:00              # maximum execution time (HH:MM:SS)
+              #SBATCH --output=/gpfswork/rech/six/uty16tp/code/big_science/logs/compute_stats_v5/%x-%j.out           # output file name
+              #SBATCH --array=1-604

+                  parser.add_argument(
+                      "--save-batch-size", type=int, required=True, help="Batch size when writing."
+                  )
+                  parser.add_argument("--use-datasets-caching", action="store_true")

	"--save-path-stats-full-json", type=str, help="Where to save the stats json."
	"--save-path-stats-full-json", type=str, required=True, help="Where to save the stats json."

+                  ]
+                  ds_html = ds_html.map(
+                      get_length_text,
+                      batched=False,

add files to compute basic stats on pseudo crawl dataset by SaulLu · Pull Request #382 · bigscience-workshop/data_tooling · GitHub

Uh oh!

add files to compute basic stats on pseudo crawl dataset - #382

add files to compute basic stats on pseudo crawl dataset#382SaulLu wants to merge 8 commits intobigscience-workshop:masterbigscience-workshop/data_tooling:masterfrom SaulLu:compute-statisticsSaulLu/data_tooling:compute-statisticsCopy head branch name to clipboard

Conversation

SaulLu commented Feb 1, 2022

Uh oh!

thomasw21 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HugoLaurencon commented Feb 8, 2022

Uh oh!

SaulLu commented Feb 8, 2022

Uh oh!

HugoLaurencon commented Feb 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

add files to compute basic stats on pseudo crawl dataset#382
SaulLu wants to merge 8 commits into
bigscience-workshop:masterfrom
SaulLu:compute-statistics